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The  purpose  of  this  study  was  two-fold:  (1)  the  development  of  models  for 
estimating  the  cost  of  Reliability  and  Maintainability  (R&M)  program  tasks  and  (2)  the 
determination  of  the  feasibility  of  developing  eost/benefit  ratios  for  each  sueh  task  or 
set  of  tasks. 

A  data  base  consisting  of  historical  project  file  information  was  established  from 
which  a  large  set  of  statistically  valid  models  for  estimating  R&M  task  labor  was 
developed.  Military  electronics  projects  of  varying  size  and  complexity  were  provided 
by  the  Hughes  data  base.  The  cost  data  consisted  of  approximately  40,000  weekly  labor 
records  which  were  categorized  by  R&M  tasks  similar  to  those  defined  in  MIL-STD's 
785  :uid  470. 

To  determine  tiutgible  benefits  from  conducting  R/M  t asks,  four  approach.es  were 
investigated.  These  were:  (1)  direct  assessment  of  the  benefit  or  gain,  (2)  ;issessment 
bv  case  study,  (2)  assessment  by  expert  opinion,  and  (4)  regression  analysis  of  observed 
benefits  with  R/M  task  application  characteristics.  Several  of  these  approaches  are 
recommended  for  future  study. 

S.2  Overview  of  Study  Results 

A  method  of  estimating  R&M  task  cost  (i.e.,  labor  hours)  based  on  data  that  is 
normally  available  during  the  early  program  planning  stages  has  been  developed.  The 
method  consists  of  a  set  of  seven  cost -estimating-relationships  (CER’s);  one  general 
relationship  for  estimating  total  R/M  program  costs  and  six  relationships  specific  to 
individual  R/M  task  areas.  These  CER's  provide  a  tool  for  comparing  alternate  R&M 
program  costs  (i.e.,  different  tailorings  of  MIE-STDs  785  and  470),  and  give 
rules-of-thumb  for  estimating  what  a  specific  R&M  program  should  cost  under  stated 
conditions.  The  CER’s  were  derived  from  data  on  the  projects  using  multiple  line  r 
regression  (MLR)  techniques.  Many  of  the  MIL-STD  785  and  470  tasks  required 
grouping  in  order  to  provide  sufficient  data  to  generate  statistically  significant  CER’s. 
The  data  btise  and  study  results  are  summar  ized  in  the  following  paragraphs  with  details 
provided  in  the  body  of  the  report. 

The  weekly  labor  record  format  used  in  the  data  base  had  separate  codes  for  each 
of  the  Mil. -STD-785  uid  470  tasks  which  could  be  computer  sorted  by  project.  A 
"point"  in  the  data  base  consists  of  a  project  identifier,  task  description  data  for  each 
R/M  task  required  by  the  program,  and  the  corresponding  labor  appenditures  for  each 
task.  In  itnuiy  cases,  significant  statistical  correlations  were  often  achieved  only  when 
R-labor  charges  we'<>  combined  with  M-labor  charges  for  the  same  type  of  tasks,  such 
as  the  preparation  of  program  plans.  Similarly,  higher  correlation  was  also  achieved 
when  like  tasks  were  combined;  for  example,  the  E  RAC  AS  effort  (Task  104)  combines 
naturally  with  Eailure  Review  board  effort  (Task  105).  Moreover,  a  number  of  R&M 
•asks  tre  classified  under  the  general  heading  of  project  management.  These  include 
such  tasks  as  administration  of  the  R&M  programs  by  the  lead  engineer  (not  identified 
as  a  Mil. -STD  task),  subcontractor/vendor  monitoring  and  control  (Task  102),  ;uid 
participation  in  program  reviews  f  lask  103). 
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Twenty-three  Hughes  projects  in  various  stages  of  development,  production,  and  of 
field  usage  were  surveyed  and  determined  to  have  sufficient  data  history  to  he  included 
in  the  data  base.  Specific  data  points  used  to  develop  the  models  (i.e.,  labor  hours  mid 
associated  systems  and  task  application  characteristics)  were  obtained  from  three 
sources:  (1)  computerized  historical  labor  records  on  specific  R&M  tasks  that  had  been 
maintained  across  each  of  the  projects;  (2)  equipment/descriptive  data  from  available 
project  files;  and  (3)  interviews  with  the  reliability  and  maintainability  engineers 
responsible  for  conducting  the  various  R/M  tasks. 


Cost  estimating  relationships  (CER's)  were  developed  based  on  standard  multiple 
linear  regression  (MLR)  techniques.  The  fitting  procedure  ensured  inclusion  of  all 
variables  which  had  predicative  importance.  \t  every  stage  of  the  CER  development, 
engineers  experienced  in  pricing  R&M  tasks  as  well  as  the  project  responsible  R/M 
engineers  were  consulted  to  ensure  that  the  estimating  techniques  were  reasonable, 
plausible,  ;uid  usable  during  early  task  planning  phases. 


A  single  CT.R  was  developed  to  estimate  the  total  R/M  program  costs.  The  model 
was  designed  for  ease  of  use  while  retaining  a  sufficient  number  of  task  description 
piU'ameters  to  provide  reasonable  estimates.  More  accurate  models  of  individual  R/M 
tasks  or  groups  of  tasks  were  also  developed,  and  these  CLR's  are  recommended 
whenever  the  appropriate  level  of  project  detail  is  available  to  meet  the  input  data 
requirements  of  the  model. 


The  CER  for  total  R/M  program  costs  is  described  in  Section  1.3  of  this  report,  and 
the  CLR’s  for  each  R/M  task  area  are  described  in  Section  3.0.  Within  Section  3.0, 
each  CLR  is  completely  documented.  The  information  on  each  CER  includes:  (1)  a 
definition  of  the  input  data  requirements  with  ranges  of  application  for  each  of  the 
model  parameters;  (2)  detailed  CER  model  description  including  statistical  fit 
information  and  examples  of  how  to  apply  the  models;  and  (3)  tabled  values  of  selected 
parameter  ranges  using  the  model  with  the  "best-tit"  statistics. 


The  study  data  base  used  to  generate  the  CER's  can  be  expanded  by  adding  project 
data  points  to  the  input  data  given  in  Appendix  B,  adjusting  the  CER  coefficients 
through  regression,  and  adhering  to  the  constraints  on  the  v.triables  given  under  the 
appropriate  subsections  of  3.0. 


S.3  Recommendations  for  Future  Studv 


The  primary  objective  intended  for  this  study  was  to  establish  CER's  for  the  R/M 
program  tasks.  To  determine  the  benefits  gained  from  the  expenditures  on  these  R/M 
tasks  is  a  difficult  but  natural  follow-on  study.  The  investigation  conducted  in  this 
study  to  determine  the  feasibility  of  deriving  measurable  benefits  from  conducting 
selected  R/M  Tasks  (see  Section  4.0)  indicated  that  the  multiple  linear  regression  (MLR) 


•MW!yy.  vy.e 


ft 

Ft! 


JvV A«/>" 


analysis  approach  is  the  most  promising.  The  MLR  approach  would  he  consistent  with  the 
('LR  development  used  in  this  study  and  has  the  advantage  of  being  the  most  objective  of 
the  approaches  investigated.  The  development  of  a  benefits  model  using  MLR,  however, 
would  require  a  much  larger  sample  size.  A  minimum  sample  size  of  30  systems  is 
recommended.  To  provide  supporting  information  for  this  development,  it  is  also 
recommended  that  industry  questionnaires  and  case  studies,  as  referenced  in  this  report, 
be  included  in  ;my  future  effort. 


In  addition,  the  life  cycle  benefits  of  an  R/M  program  should  be  considered. 
Improvements  in  hardware  R/M  resulting  from  the  implementation  of  an  R/M  program 
iiuuiifest  themselves  as  cost  savings  derived  in  reduced  spares  and  maintenance 
manpower  or  in  increased  operational  readiness  and  could  have  a  substantial  impact 
over  the  life  of  the  system.  Only  a  small  gain  in  actual  MTBF  per  system,  for  example, 
would  result  in  a  large  life  cycle  cost  reduction  in  the  maintenance  support  areas  for  a 
large  deployment  of  such  systems. 
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1.0  SUMMARY  OF  STUDY  RESULTS 


1.1  Introduction. 

1.1.1  Purpose.  The  objectives  of  this  study  were  the  development  of  quantitative 
or  heuristic  models  for  estimating  on  an  individual  or  composite  task  basis,  the  cost  of 
Reliability  and  Maintainability  (R&M)  programs  for  electronic  equipment  and  the 
determination  of  the  feasibility  of  developing  cost/benefit  ratios  for  each  such  task  or 
set  of  tasks. 

1.1.2  Background.  MIL-STD-785  and  MIL-STD-470  define  the  necessary  R&M 
activities  needed  to  ensure  equipment/system  designs  comply  with  specified 
Government  requirements.  Currently,  problems  exist  in  both  (1)  defining  the  cost  of 
R&M  programs  in  terms  of  the  individual  program  tasks  and  (2)  determining  the 
cost/benefit  associated  with  each  task.  (In  other  words,  how  much  reliability  or 
maintainability  can  a  given  task  buy?)  Previous  models,  guidelines,  data  and  analyses 
are  no  longer  appropriate.  Technology  has  changed  and  emphasis  in  this  area  has 
shifted  to  the  use  of  tailored,  as  opposed  to  blanket,  R&M  programs.  Both  Government 
and  industry  require  information  and  guidance  such  that  visibility  and  insight  may  be 
gained  as  to  the  cost  ramifications  and  cost  effectiveness  of  R&M  programs. 

1.1.3  General  Approach  to  the  Study.  An  extensive  in-house  data  base  was 
accessed  in  order  to  establish  a  large  set  of  statistically  valid  models  for  estimating 
R&M  task  labor.  A  minimum  of  eight  different  military  equipment/system  programs 
were  required  by  the  study  Statement  of  Work  but  it  was  believed  that  a  much  larger 
data  base  was  needed  to  provide  a  statistically  valid  result.  A  total  of  23  projects  of 
varying  size  and  complexity  were  provided  by  the  Hughes  data  base.  The  data  base 
consisted  of  approximately  40,000  weekly  labor  records  which  were  categorized  by 
R&M  task.  Reliability  engineers  within  the  Systems  Effectiveness  laboratory  reviewed 
the  data  and  researched  any  anomalies  discovered  in  the  data.  A  multiple  linear 
regression  approach  was  used  on  a  set  of  key  system  and  task  application  variables. 

1.2  R/M  Cost  Models,  Ground  Rules  and  Assumptions.  In  determining  effective 
cost-estimating-relationships  (CER's),  it  is  essential  to  ensure  that  all  variables  which 
have  a  significant  influence  on  R&M  task  cost  (i.e.,  labor  hours)  are  included  in  the 
relationship.  Moveover,  the  input  data  needed  to  use  the  CER's  should  consist  of 
information  typically  available  prior  to  the  performance  of  the  associated  R&M  tasks. 

The  general  forms  of  the  CER's  are  given  by: 

(1.2-1)  T  =  C  Q+  C  j  Zj  +  C2  Z  2+  ...  +  CpZp  ,  and 

C  C  C 

(1.2-2)  T  =  C0Zj  Z^  •  •  •  Zpr 

where  1  is  the  dependent  cost  variable  expressed  in  labor  hours  to  perform  the  specified 
R/M  task  or  set  of  tasks;  the  Zj  are  the  independent  variables  representing  system  and 
task  application  characteristics,  or  functions  thereof,  which  have  significant  correlation 
with  T;  and  the  Cj  are  constants  of  tiie  CER  determined  by  the  regression  fit.  Equation 
(1.2-2)  is  not  linear  in  the  independent  variables  Zj  and,  therefore,  must  be  transformed 
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to  linear  form  before  MLR  techniques  can  be  applied.  Taking  the  natural  logarithms  of 
(1.2-2),  results  in  the  following  linear  form: 

(1.2-3)  In  T  =  In  C0  +  C i  In  Z]  +  C2  In  Z 2  +  •  •  •  +  CP  In  Zp 

All  discussions  in  this  report  referring  to  MLR  models,  fit  characteristics  (i.e.,  R  K 
regression  error  etc.)  and  associated  assumptions  pertain  to  equations  (1.2-1 )  and 
(1.2-3).  A  detailed  discussion  of  the  MLR  model  development  process  is  provided  in 
Appendix  A.  A  brief  summary  of  the  model  assumptions  and  ground  rules  is  given  below: 
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1.  The  dependent  variable  in  (1.2-1)  or  (1.2-3)  is  a  linear  combination  of  p 
independent  variables.  In  matrix  notation  the  MLR  model  is  represented 
in  the  form: 

Y  =  XB  +  e 


where: 

Y  =  (n  x  1)  vector  of  observations  of  the  dependent  variable: 

(T j,  T 9,  ...,  Tn)  for  (1.2-1)  and  (InT j,  InT 2,  ...,  lnTn) 
for  (1.2-3). 

X  =  (n  x  p)  matrix  of  observations  of  the  independent  variables; 

X  j  j  =  Z  j  j  f  or  (1.2-1)  and  X  jj  =  InZjj  for  ( 1 . 2-3),  1  i  <_  n 
and  1  ^_j  <  p. 

R  =  (p  x  FT  vector  of  constants  (CER  coefficients)  to  be  estimated, 
e  =  (n  x  1)  vector  of  errors. 

2.  The  elements  ej,  I<j<_n,  of  the  vector  e  represent  values  of  a  normally 
distributed  random  variable. 

3.  E(e)  =  (o)  and  Var  (e)  =  I  ,  where  I  is  the  identity  matrix,  so  that  the 
elements  of  e  are  assumed  to  be  uncorrelated. 

Table  1 .2-1  identifies  the  t’E  R 's  and  associated  MFL-STH-  78^470  task  (or  group  of 
tasks)  and  provides  some  general  statistics  on  how  well  the  model  fits  observed  data. 
R  ^  values  greater  than  0.9  are  generally  considered  excellent  fits.  The  last  column 
references  the  report  paragraph  which  defines  the  input  data  requirements,  the  TER 
model,  and  example  applications.  A  summary-level  model  was  also  developed  which 
estimates  the  total  labor  hours  expended  in  an  R/M  program  (see  1.3). 


Although  the  weekly  labor  record  format  had  separate  codes  for  each  of  the 
MIL-STP-785  and  470  tasks,  significant  correlations  were  often  achieved  only  when 
R  -labor  charges  were  combined  with  M-Iabor  charges  for  the  same  type  of  tasks  (e.g., 
Tasks  101,  201,  202  and  203).  Similarly,  higher  correlation  was  also  achieved  when  like 
t asks  were  combined  (e.g.,  104  combined  with  105,  201  combined  with  202  and  3 0 1 ,  303 
and  304  combined). 


\  number  of  RAM  tasks  are  classified  under  the  general  heading  of  project 
management  (see  Table  1.2-1).  These  tasks  include:  administration  of  the  RAM 
programs  by  the  lead  engineer;  subcontractor/ vendor  monitoring  and  control  (Task  102): 
participation  in  program  review's  (Task  103):  identification  and  controls  for  reliability- 
critical  items  ('['ask  208);  training  of  newly  assigned  R/M  engineers:  and  general  RAM 
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support  to  the  project  systems  and  design  engineers  not  covered  by  other  R& M  tasks. 
For  the  systems  in  the  data  base,  project  management  accounted  for  approximately 
16%  of  the  total  R&M  task  effort. 


Parts  programs  (Task  207)  can  have  a  significant  cost  impact.  This  is  especially 
the  case  with  high  reliability  state-of-the-art  designs  where  the  potential  for  large 
numbers  of  non-standard  parts  exists.  However,  project-related  labor  records  for 
implementing  and  conducting  a  parts  program  were  not  maintained  in  sufficient  detail 
to  develop  a  CER.  General  rules-of-thumb  based  on  Hughes  standard  policy  on 
selection  control  for  a  parts  program  in  accordance  with  MIL-STD-965,  Procedure  1, 
are  given  below: 


a. 

Program  Plan  Preparation 

80-1  20  hours 

b. 

Non-standard  parts  justification 
submittal  and  follow-up 

5-8  hours  per  part 

c. 

Specification  preparation  (for 
non-standard  parts  procurement) 

32-48  hours  per  part 

If  special  reliability  screening  is  employed  to  upgrade  a  non-standard  part,  the  recurring 
test  cost  could  add  2  to  5  dollars  (1986)  to  the  price  of  a  complex  integrated  circuit. 
Procedure  2  of  MIL-STD-965  would  add  the  cost  of  the  parts  control  board  which  is 
dependent  on  how  often  the  board  convenes,  location,  etc. 

The  remaining  MIL-STD-785  and  470  tasks  are  either  included  in  several  of  the 
tasks  modeled  or  had  insufficient  data  as  noted  in  Table  1.2-1. 

1.3  General  R/M  Cost  Model.  A  single  overall  CER  was  developed  to  estimate  the 
total  R/M  program  costs  (i.e.,  as  defined  in  Section  3.0).  The  model  was  designed  to  be 
easy  to  use  during  the  early  R/M  program  planning  stages  while  retaining  enough  R/M 
task  descriptive  parameters  to  provide  reasonable  estimates. 

For  the  derivation  of  this  model,  task  information  from  each  project  in  the  data 
base  was  assembled  into  a  single  data  set.  Tasks  not  performed  on  particular  projects 
have  corresponding  entries  equal  to  zero.  This  yielded  a  data  set  with  23  projects  in 
which  one  or  more  MIL-STD-785/470  tasks  were  performed.  The  descriptions  of  the 
program  factors  (independent  variables)  are  contained  in  the  appropriate  sub-section  of 
Section  3.0. 

There  were  several  considerations  when  exploring  the  form  of  the  model:  ease  of 
use  and  consistency  of  estimates  were  foremost.  Simply  summing  the  results  of  the 
individual  models  produced  good  estimates  but  an  unwieldly  equation.  This  general 
form  was  retained  but  exponents  were  simplified  and  parameters  were  deleted 
whenever  the  fit  did  not  suffer  appreciably.  The  MLR  program  output  can  be  found  in 
Table  B.7.2-7.  An  R^  value  of  0.85  was  achieved  for  a  model  with  only  one  descriptor 
for  most  of  the  R/M  tasks.  The  final  CER  is  given  below,  where  each  term  in  the 
equation  corresponds  to  an  R/M  task  or  set  of  tasks  as  indicated: 

(1.3-1)  TR/M  =  2.73  (NOT)2  +  8.25  (DOI)2  +  4.05  (MAC)2  (NOU) 


i 


+  4.54  (LOD)2  (RF)2  (POC)  +17.79  (NOU')  +  182.07  (HC) 

Tr/\]  =  Total  cost  (labor  hours)  of  performing  the  indicated  tasks. 

NOT  =  Number  of  R/M  tasks  to  be  conducted.  Relates  to  the  task  of  developing 
an  R/M  program  plan  (Task  101). 

001  =  Duration  of  FR  ACAS  implementation  (Tasks  104  and  1 05). 

MAC  =  Modeling  and  Allocation  complexity  factor  for  R/M  Tasks  201  and  20  2 

(see  3.3.1  for  scaling). 

NOU  =  Number  of  unique  items  in  the  allocation  process  (Tasks  201  and  202). 

LOD  =  Level  of  detail  of  the  R/M  prediction,  Task  203  (see  3.4.1  for  scaling). 

R  F  =  Prediction  reporting  formality,  Task  203  (see  3.4.1  for  scaling). 

POC  =  Percentage  of  commercial  equipment  used  in  the  system,  Task  203  (see 

3.4.1  for  scaling). 

NOU*  =  Number  of  unique  items  requiring  an  FMECA  (Task  204). 

HC  =  Hardware  complexity  in  terms  of  total  electronic  part  count,  Tasks  301, 

303  and  304  (see  Table  3.2-2  for  sealing). 

Tasks  not  performed  in  a  given  program  are  accounted  for  by  substituting  zero  for 
the  appropriate  independent  variables.  Project  management  can  be  included  as  a  16% 
factor  of  the  total  computed  task  effort,  and  a  parts  program  cost  (based  on  procedure 
1  of  MIL-STD-965)  can  be  added  using  the  above  rules-of-thumb  (see  1.2). 

As  an  example  of  use  of  the  above  model,  Project  204  data  is  employed  (refer  to 
Tables  B.fi-1  through  B. fir-6).  The  pertinent  descriptors  for  the  R/M  are  given  below: 


NOT 

=  o, 

LOD 

=  3 

DOI 

=  36, 

RF 

=  2 

MAC 

=  1, 

POC 

=  4 

NOU 

=  445, 

NOU' 

HC 

=  3 
=  3 

Substituting  these  values  into  (1.3-1)  results  in: 
Tr/M  =  2.73  (0)  +  8.25  (36) 2 
+  4.05  (1)2(445) 

+  4.54(3)2(2)2(4) 
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-  13748  Hours  (or  15,948  hours  including  project  management) 

The  observed  number  of  labor  hours  in  conducting  R/M  tasks  for  Project  204  was 
1 2,788  (without  project  management),  a  difference  of  approximately  8%  from  the 
estimate. 

The  simp’ified  model  represented  in  (1.3-1)  gives  reasonable  labor  estimates  for  an 
R/M  program.  However,  for  more  accurate  estimates  of  an  individual  task,  one  of  the 
models  defined  in  Section  3.0  for  that  specific  task  is  recommended. 

1.4  Investigation  of  Derived  Benefits  from  R/M  Tasks.  Four  approaches  to 
deriving  a  means  of  measuring  tangible  benefits  from  conducting  R/M  tasks  were 
investigated.  Section  4.0  provides  the  details  of  this  investigation  and  the  results  are 
summarized  below: 

a.  Direct  assessment  of  the  benefit  or  gain  from  performing  appropriately 
grouped  R&M  tasks  requires  an  inordinate  amount  of  detailed  engineering 
data.  In  addition,  this  data  is  generally  too  subjective  (i.e.,  as  to  what 
portion  of  a  change  was  due  to  R/M)  and  unique  to  a  company's  way  of 
doing  business  to  have  broad-based  applications. 

b.  Task  assessment  by  case  study  is  a  useful,  but  generic,  and  often 
subjective,  appraisal  of  R<3cM  benefits  best  suited  to  support  more  detailed 
analyses.  Six  Institute  for  Defense  Analysis  (IDA)  studies  were  examined 
showing  the  key  benefit  areas  to  be:  Test-analyze-and-fix  (TAAF), 
FRACAS,  worst-case/thermal  analyses,  stress  screening  and  R-growth. 
However,  the  studies  did  not  assign  benefits  to  specific  task  areas. 

c.  Task  assessment  by  expert  opinion  suffers  from  the  same  problems  as  in  b.f 
in  that  the  responses  to  a  quantitive  survey  questionnaire  are 
inadequate  for  statistical  analysis  and  qualitative  questionnaires  only 
provide  supportive  information. 

d.  Regression  analysis  of  observed  R/M  benefits  with  respect  to  program  and 
R/M  task  application  characteristics  (as  established  in  the  CER 
development)  provides  an  objective  result  but  requires  a  large  sample 
size.  This  approach  only  requires  an  estimate  of  the  total  benefit  (as 
opposed  to  task-by-task  estimate  of  benefit)  and  can  be  compared  to 
other  more  subjective  results  for  consistency. 
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TABLE  1.2-1.  K/iVi  TASK  COST  ESTIMATING  RELATIONSHIPS 
Detailed  descriptions  and  example  computations  are 
provided  in  the  referenced  paragraphs. 


CER 

Description 

MIL-STD-785/470 
Task  Reference 

No.  of 
Data 
Points 

Degrees 

of 

Freedom* 

R2 

Value* 

Comments/ 

Reference 

Paragraph 

R&M  Program  Plan 

101R/M 

10 

6 

0.97 

3.1 

Monitor  Control 
of  Subcontractor 
Suppliers 

102R/M 

Included  under 
>R/M  project 
i  management. 

Program  Reviews 

103R/M 

- 

- 

- 

> 

FRACAS/Failure 
Review  Board 

104R/M,  105R 

13 

8 

0.99 

3.2 

R&M  Modeling 
&  Allocations 

201  R/M,202R/M 

8 

3 

0.95 

3.3 

R&M  Predictions 

203ft/ M 

16 

10 

0.97 

3.4 

FMECA 

204R/M 

6 

2 

0.99 

3.5 

Sneak  Circuit 

Analysis  Maintenance 

205R 

- 

- 

- 

Insufficient 

Information 

Maintenance  Analysis 

205M 

“ 

Included  in 

Tasks  201,  202, 
203  and  204 

Electronic  Parts/ 
Circuit  Tolerance 
Analysis 

206R 

— 

“ 

Included  in 

203 

Maintenance  Design 
Criteria 

206M 

- 

- 

- 

Included  in 

101 

Parts  Program 

207R 

Hughes  policy 
on  selection 
and  control. 

(see  1 . 2) 

Design  guides 
included  in  101. 

Preparation  of  Inputs 
to  the  Detailed  Main¬ 
tenance  Plan  and 

LSA 

207M 

Included  in 

203  and  204 

TABLE  1.2-1.  R/M  TASK  COST  ESTIMATING  RELATIONSHIPS  (Continued) 


CER 

Description 

M1L-STD-785/470 
Task  Reference 

No.  of 
Data 
Points 

Degrees 

of 

Freedom  * 

R2 

Value* 

Comments/ 

Reference 

Paragraph 

Reliability  Critical 
Items 

208R 

- 

- 

- 

Included  in 

R/M  project 
management 

Effects  of  Functional 
Testing,  Storage, 
Packaging,  Trans¬ 
portation  and 
Maintenance 

209R 

No  infor¬ 
mation 

Reliability  Testing 

301,  303,  304 

7 

3 

0.95 

3.6 

Maintainability 

Demonstration 

301M 

— 

— 

— 

Insufficient 

Information 

Reliability  Develop¬ 
ment/Growth  Test 
Program 

302R 

— 

— 

Included  in 
104/105 

a 
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2.1  Sources  of  Data.  Thirty-three  Hughes  systems  in  various  stages  of 
development,  production  and  of  field  usage  were  surveyed.  Twenty-three  of  these 
systems  were  eventually  determined  to  have  sufficient  data  history  to  be  included  in 
the  data  base  (i.e.,  valid  labor  accounting  records  over  the  period  in  which  the  R/M 
tasks  were  conducted).  The  final  data  base  provides  a  sufficient  size  and  range  in  the 
values  of  the  independent  variables  so  that  the  CER's  will  have  a  wide  range  of 
application  to  different  types  of  systems/equipment.  Individual  equipment  as  well  as 
complete  systems  are  contained  in  the  data  base  representing  radar,  communications, 
display  and  data  processing  technologies.  All  data  points  represent  military  systems/ 
equipment  from  USAF,  Army  and  Navy  contracts  and  reflect  a  wide  range  of  operating 
environments  (Gp,  Gjy],  Ng,  AyF  and  Ugp  as  defined  in  MIL-HDBK-217). 


Specific  data  points  used  to  develop  the  models  (i.e.,  labor  hours  and  associated 
system  and  task  application  characteristics)  were  obtained  from:  (1)  computerized 
historical  R&M  tasks  labor  records  that  had  been  maintained  across  each  of  the  above 
projects;  (2)  equipment/descriptive  data  from  available  project  files;  and  (3)  interviews 
with  the  reliability  and  maintainability  engineers  responsible  for  conducting  the  various 
R/M  tasks.  Most  of  the  project  descriptive  data  came  from  interviews  and  subsequent 
follow-ups.  In  order  to  maintain  uniformity  across  the  project  data  (i.e.,  with  respect 
to  definitions  of  variables,  scaling  of  qualitative  inputs,  general  assumptions,  etc.),  a 
questionnaire  format  was  developed  and  filled  out  during  each  interview.  The  input 
data  for  each  CER  is  provided  in  Appendix  B  by  project  (see  Tables  B.6-1  through  B.6-6). 


2.2  Data  Processing  Methodology.  CER's  were  developed  based  on  standard 
multiple  linear  regression  (MLR)  techniques.  The  fitting  procedure  ensured  inclusion  of 
all  variables  which  had  predictive  importance.  Where  appropriate,  alternate  CER's  are 
provided  along  with  a  best  fitting  CER  for  each  task.  The  alternates  have  fewer 
variables  requiring  input  data  but  do  not  provide  as  good  an  estimate  of  task  cost. 
Measures  of  their  relative  fit  (R2  and  F-test  values)  in  comparison  to  the  best  equation 
are  provided  in  the  detailed  description  of  each  CER  (Section  3.0.)  At  every  stage  of 
the  CER  development,  engineers  experienced  in  pricing  R&M  tasks  as  well  as  the 
responsible  R/M  engineers  were  consulted  to  ensure  that  the  estimating  techniques 
were  reasonable,  plausible,  and  usable  during  early  task  planning  phases.  The  data 
processing  method  is  illustrated  in  Figure  2.2-1  and  described  below.  A  more  detailed 
discussion  of  the  analysis  tools  and  evaluation  criteria  employed  in  developing  the 
CER's  is  given  in  Appendix  A. 


Starting  with  a  large  set  of  candidate  independent  variables,  a  questionnaire  was 
developed  which  categorized  these  variables  bv  appropriate  R<3cM  tasks.  The  first 
"screen"  in  the  process  was  to  eliminate  those  variables  which  did  not  have  sufficient 
data  for  statistical  analysis.  The  remaining  variables  were  then  compiled  into  the  CER 
data  base  together  with  the  corresponding  RAM  task  labor  hours.  The  second  screen 
eliminated  those  independent  variables  and  data  points  which  caused  "defects"  in  the 
CER: 


Form  of  CER  -  The  CER  was  considered  defective  if  one  or  more 
independent  variables  had  negative  coefficients  (e.g.,  a  minor  variable  had 
inverse  correlation  with  one  or  more  variables  already  included  at  a 
previous  step  in  the  regression). 
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DEFINE  CANDIDATE 

independent 

VARIABLES 


DEVELOP 
QUESTIONNAIRE 
AROUND  CANDIDATE 
VARIABLES 


CONDUCT  INTERVIEWS 
OBTAIN  DATA  FROM 
PROJECT  FILES 


OBTAIN  PROJECT 
LABOR  HOURS 
FROM  COMPUTER 
FILES 


UPDATE  DATA  BASE 


NEW  MODEL 


COMPILE  CER 

data  base 

GENE 

CER 

USINC 

RATE 

MLR 

CER  CHECK 
CRITERIA  : 

•  FORM  OF  RELATIONSHIP 

•  r2  value 

•  f-test 

•  MLR  ASSUMPTIONS 


DOCUMENT 
CER  FOR  FINAL 
REPORT 


DELETE 
DATA  POINT 


DELETE 

VARIABLE 


•  R-  value/  K-Test  -  The  variable  either  did  not  have  correlation  with  cost 
or  was  highly  correlated  with  another  (better)  variable. 

•  MLR  assumptions  -  The  dependent  or  independent  variable  has  correlation 
with  the  residual  error  (i.eM  violates  the  assumptions  of  normality). 

•  Outliers  -  The  data  point  is  not  typical  of  the  rest  of  the  data  base.  This 
type  of  defect  was  carefully  examined  to  determine  possible  cause  (e.g.,  a 
unique  program  characteristic  which  could  be  isolated  from  the  data)  and 
only  removed  if  it  could  not  be  explained  or  was  determined  to  be  an 
uncorrectable  error  in  data  recording. 

'2.3  Program  Characteristics  Impacting  R/M  Task  Costs.  Tables  2.3-1,  -2  and  -3 
provide  a  compilation  of  the  variables  which  were  found  to  have  predictive  value  in 
estimating  RAM  task  labor.  The  tables  partition  the  variables  as  to  whether  they  are 
program  related  (i.e.,  require  general  information  on  the  test  schedule  and 
implementation  of  the  RAM  program),  system  related  (i.e.,  require  data  on  the  number 
and  complexity  of  the  system),  or  task  application  related.  The  definition  and  range  of 
values  of  each  variable  are  provided  in  the  paragraphs  referenced  in  the  tables.  In 
addition,  the  R/M  tasks  that  each  variable  is  applicable  to  are  noted  in  the  last  column 
of  these  tables. 
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TABLE  2.3-1.  R&M  PROGRAM  RELATED  VARIABLES 


Description 


Number  of  R/M  Tasks 


Reporting  Formality 


Duration  of  Implementatio 
(FRACAS) 


Average  Number  of  Units 
in  FRACAS  Program 


Number  of  Environmental 
Screens 


Requirements  Complexity 


Reliability  Qualification 
Testing 


Production  Reliability 
Acceptance  Testing 


Symbol  |  Definition  |  Applicable  R/M  Tasks 


101  R/M 


1 04  R/M,  105R 


104R/M,  1 05R 


301  R,  303R,  304R 


201 R/M,  202R/M 


301 R,  303R,  304R 


301 R,  303R,  304R 


NOT 

3.1.1 

RF 

3.1.1,  3.2.1, 
3.4.1 

DOI 

3.2.1 

ANU 

3.2.1 

NOS 

3.6.1 

RC 

3.3.1 

RQT 

3.6.1 

PRAT 

3.6.1 

TABLE  2.3-2.  SYSTEM  RELATED  VARIABLES 


Description 
Hardware  Complexity 


Modeling/  Allocation 
Complexity 


Number  of  Unique  Items 


Percent  of  Commercial 
Equipment 


Symbol 

Definition 

HC 

3.2.1 

MAC 

3.3.1 

NOU 

3.3.1,  3.5.1 

RNR 

3.3.1 

POC 

3.4.1 

Applicable  R/M  Tasks 


104R/M,  105R,  201  R/M, 
202R/M,  203R/M,  204R/M, 
301R,  303 R,  304R 


201  R/M,  202 R/M 


203  R/M 


203R/M 


3.0  DETAILED  DESCRIPTIONS  OF  R&M  COST  MODELS 


3.1  R&M  Program  Plan  (Task  101).  The  effort  required  to  develop  a  program  plan 
depends  on  the  number  of  tasks  included  in  the  plan,  the  depth  of  coverage,  and  whether 
the  plan  requires  formal  customer  approval  or  not. 

Since  the  majority  of  R  and  M  tasks  are  interrelated  with  overlapping  "plan" 
coverage  in  most  cases,  it  is  usually  not  practical  to  plan-out  the  R-program  separately 
from  the  M-program.  This  perception  was  reflected  in  the  labor  accounting  records, 
where  the  split  in  labor  charges  between  the  R-program  plan  and  M-program  plan 
developments  appeared  to  be  arbitrary.  Accordingly,  when  the  labor  charges  were  kept 
separate,  very  little  correlation  with  the  above  factors  was  evident  but  when  the  labor 
charges  were  combined,  a  very  high  correlation  was  achieved  (see  Table  3. 1.2-1).  The 
validity  of  combining  the  labor  charges  was  also  justified  in  discussions  with  the  Lead 
Engineers  responsible  for  the  R<5cM  activities  on  many  of  the  programs  in  the  data  base. 
These  Lead  Engineers  were  typically  responsible  for  the  R&M  activities  on  many  of  the 
programs  in  the  data  base.  Lead  Engineers  were  also  typically  responsible  for  writing 
the  R  and  M  program  plans  (usually  a  single  integrated  plan)  with  inputs  from  assigned 
personnel. 

3.1.1  Input  Data  Requirements.  Table  3.1-1  gives  the  key  program  factors 
(independent  variables)  determined  to  have  significant  correlation  with  the  cost  of 
developing  an  "integrated"  R<5cM  program  plan  and  provides  range  of  applicability  for 
each  factor. 

The  following  paragraphs  discuss  the  meaning  and  any  qualifying  constraints  on 
their  use  in  the  CER  models  (3.1.2): 

Depth  of  Coverage  (DOC).  This  variable  represents  the  average  amount  of  detail 
that  is  provided  for  each  R<5cM  task  required  by  the  contract.  A  small  amount  of  detail 
(DOC=l)  includes  only  a  descriptive  paragraph  with  referenced  military  standards, 
associated  schedule  milestones  and  an  organization  structure  showing  the  relationship 
between  R&M  and  project  management.  A  medium  amount  of  detail  (DOC=2)  adds  the 
related  procedures  for  accomplishing  the  task,  describes  the  interfaces  (inputs/outputs) 
from  and/or  to  other  disciplines  (design  engineering,  manufacturing,  QA,  logistics,  etc.), 
and  a  large  amount  of  detail  (DOC=3)  adds  R<5cM  design  guides  and  checklists. 


TABLE  3.1-1.  PROGRAM  FACTORS  CORRELATED  WITH 
R<ScM  PROGRAM  PLAN  DEVELOPMENT 


Factor 

Range  of  Application 

Minimum  ' 

Maximum 

Units 

Depth  of  Coverage  (DOC) 

1 

3 

Scale 

Number  of  R&M  Tasks  (NOT) 

4 

22 

- 

Reporting  Formality  (RF) 

1 

2 

Scale 

Number  of  RAM  Tasks  (NOT).  This  variable  represents  the  total  number  of  RAM 
tasks  as  defined  by  MIL-STP's  785  and  470. 

Reporting  Formality  CRF) .  RiiM  plans  can  cither  be  formal  fRE-l),  requiring 
review  and  approval  action  from  the  customer,  or  informal  fRI:=2'),  requiring  only 
an  internal  program  management  review.  In  cither  case,  the  R'iM  plan(s)  represent 
a  contract  to  perform  the  stated  R'iM  tashs  in  the  manner  described  and,  therefore 
should  be  carefully  written.  However,  in  a  formal  fCPRl.  submittal  1  plan,  closer 
scrutiny  of  the  contents  is  made  by  both  internal  management  and  the  customer 
resulting  in  additional  labor  expended  on  subsequent  revisions. 

3.1.2  Models.  The  genera]  model  for  estimating  the  labor  to  develop  RAM 
program  plans  has  the  form: 


(3.1. 2-1) 

T]  0  1 

=  i  DOC  |l  1  (  \0  I  j  L_’  1  RF  )  L7 

where: 

T 10! 

=  total  labor (hours) 

not: 

=  depth  of  plan  coverage  (see  3.1.1) 

NOT 

=  total  number  of  RAM  tasks  required  by 
MIL-STP’s  7  85  and  4  70 

RF 

=  report  formality  (see  3.1.1) 

Gj,  ^2*  C3  =  model  coefficients  (see  Table  3. 1.2-1) 

Equation  (3. 1.2-1)  estimates  the  labor  necessary  to  produce  both  the  R  and  M  program 
plans  even  though,  for  formally  submitted  plans,  separate  CDRL’s  may  be  required.  In 
addition,  many  of  the  program  plans  in  the  data  base  contain  a  certain  amount  of  "lift" 
from  the  plans  of  prior  programs.  This  is  normal  practice  for  companies  who  do 
substantial  business  with  military  agencies.  Unfortunately,  the  amount  of  lift  used  to 
produce  a  given  plan  was  not  a  quantifiable  factor  which  could  be  incorporated  in  the 
model.  The  general  experience  of  the  RAM  engineers  interviewed,  however,  was  that  a 
typical  program  plan  consisted  of  from  40  to  60  percent  lifted  material.  Therefore,  a 
newly  developed  program  plan  with  little  or  no  lifted  material  could  take  substantially 
more  effort  then  estimated  by  (3.1. 2-1). 

Table  3. 1.2-1  provides  three  models  for  making  estimates  based  on  available  data. 
For  example,  if  a  program  requires  a  plan  which:  (1)  consists  of  10  RAM  Tasks,  (2) 
demands  a  maximum  depth  of  coverage  for  each  task,  and  (3)  involves  Government 
review  and  approval,  then  using  equation  No.  3  in  Table  3.1. 2-1  results  in  the  following 
estimate: 
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TABLE  3.1. 2-1.  HER  MODELS  FOR  R<5cM  PROGRAM  PLANS 


Equation 

No. 


Model  Coefficients  (C  j) 

P(>F) 

not' 

NOT 

RF 

- 

2.03  6 

- 

0.0001 

- 

1.829 

1.228 

0.0001 

0.532 

1.710 

1.336 

0.0001 

0.960 


0.967 


0.969 


The  last  two  columns  in  Table  3. 1.2-1  provide  a  significance  test  and  multiple 
correlation  coefficient  for  each  model.  Significance  tests  with  probabilities  less  than 
0.05  and  R-  values  of  greater  than  0.9  are  considered  good  fits.  As  can  be  seen,  all  of 
the  equations  in  the  table  are  excellent  fits.  Appendix  B  provides  the  complete 
regression  analysis  results. 

3.1.3  Cost  Tables.  Tables  3. 1.3-1  and  3. 1.3- 2  provide  a  global  interpretation  of  the 
CER  using  equation  No.  3  of  Table  3. 1.2-1.  The  other  equations  in  Table  3.1. 2-1  would, 
of  course,  result  in  a  different  set  of  values  for  estimating  task  labor. 

Table  3. 1.3-1  represents  a  breakdown  of  informal  plan  development  labor  hours  and 
Table  3. 1.3-2  the  same  breakdown  for  plans  requiring  a  CDRL  submittal.  The  plan 
depth-of-coverage(lHH')  is  typically  reflected  in  the  total  number  of  pages  included  in 
the  plan  which  is  based  on  the  amount  of  system/program  detail  that  will  be  available 
during  the  contract  period.  This  detail  normally  (but  not  always)  follows  the  type  of 
program:  C oncept  Development,  Validation,  Full-Scale  Development  (FSD)  and 
Production.  For  the  systems  in  the  data  base,  these  associations  with  DOC  are  given  in 
Table  3. 1.3-3.  If  the  contract  is  for  system  validation,  for  example,  the  program  plan 
would  typically  be  of  medium  depth  of  coverage  and  would  normally  require  from  51  to 
100  pages  for  documentation  depending  on  the  number  of  R<5rM  tasks  included.  Some 
caution  should  be  exercised  to  ensure  that  the  number  of  RicM  tasks  and  the  depth  of 
coverage  are  consistent  with  the  program  phase  (e.g.,  an  FMECA  analysis  would 
probably  not  be  included  in  a  plan  for  the  Concept  Development  Phase  since  the 
necessary  design  detail  is  not  available). 

3.2  FRACAS/FRB  (Tasks  104  and  105).  The  amount  of  effort  required  for  Failure 
Reporting  and  C orrective  Action  System  (FRACAS)  implementation  and  Failure  Review 
Board  (FRR)  participation  is  dependent  on  the  "size"  of  the  system  or  equipment  being 
developed  (measured  in  terms  of  the  number  of  active  components)  and  the  average 
number  of  units  under  test  or  in  use  over  a  specified  period  of  time.  Average  part 
quality  is  also  a  probable  FRACAS/FRB  cost  factor  since  the  use  of  low  quality  parts 
would  be  expected  to  cause  more  failure  actions  resulting  in  increased  reporting, 
reviews  and  corrective  actions.  However,  the  range  of  part  quality  in  the  data  base  was 
typically  JAN  for  semiconductors  and  from  Class  B-2  to  Class  B  for  micro-  circuits,  and 
within  this  range  part  quality  was  not  a  significant  factor.  This  fact  should  be 
considered  when  applying  the  FRACAS/FRB  CER  in  3.2.2. 
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TABLE  3. 1.3-1.  TASK  101  R  AND  VI  PROGRAM  PLAN 
INFORMAL  REPORTING  (HOURS) 


Number 
of  R/M 


Depth  of  Coverage 


Small 

Med 

16. 

23. 

51. 

74. 

103. 

148. 

168. 

242. 

TABLE  3. 1.3-2.  TASK  101  R  AND  M  PROGRAM  PLAN  -  CDRL  REQUIRED  (HOURS) 


Number 


Depth  of  Coverage 


TABLE  3. 1.3-3.  DEPTH  OF  COVERAGE  VERSUS  PROGRAM  PHASE 


Program  Phase 


Depth  of  Coverage  (TOC)  |  Range  of  Page  Count 


Concept  Development 

Small 

0 

-  5 

•y 

gjj  Validation  -  FSD 

Medium 

51 

-  1 

f-.  FSD  -  Production 

Large 

> 

The  effort  required  for  participation  in  FRB  activities  is  inseparable  from  the 
effort  required  for  failure  reporting  and  the  taking  of  appropriate  corrective  actions. 

As  a  matter  of  company  policy  at  Hughes-Fullerton,  all  deliverable  projects  require 
FRACAS  in  which  the  FRB  is  chaired  by  the  project  Technical  Director  and  co-chaired 
by  the  lead  R&M  and  Quality  Assurance  engineers. 

3.2.1  Input  Data  Requirements.  Table  3.2-1  gives  the  independent  variables  having 
significant  correlation  with  the  FRACAS/FRB  activities  and  their  ranges  of 
applicability.  The  following  paragraphs  discuss  the  meaning  of  these  variables  and  any 
qualifying  constraints  on  their  use  in  the  CER  models  (3.2.2). 


TABLE  3.2-1.  PROGRAM  FACTORS  CORRELATED 
WITH  FRACAS/FRB  ACTIVITIES 


Range  of  Application 

Factor 

Minimum 

Maximum 

Units 

Duration  of  Implementation 
(DOI) 

2.5 

38.0 

Hardware  Complexity  (HC) 

1 

3 

Average  Number  of  Units 

On  Test  Per  Month  (ANU) 

0.3* 

30.0 

*  Only  part  of  the  system  was  tested. 

Duration  of  Implementation  (DPI).  DOI  represents  the  total  calendar  time 
expressed  in  months  during  which  FRACAS  is  active.  For  the  systems  in  the  data  base, 
this  process  generally  started  with  integration  and  checkout  of  the  first  configuration 
item  (Cl)  and  continued  until  turnover  of  the  final  system  to  the  customer. 

Hardware  Complexity  (HC).  HC  is  measured  in  terms  of  the  number  of  electronic 
parts,  (i.e.,  hybrids,  integrated  circuits  semi-conductors  resistors  and  capacitors.)  The 
count  should  include  all  active  redundant  units  and  exclude  "cold"  standby  units  that  are 
part  of  the  system  but  are  not  under  power  until  needed  (i.e.,  switched  in).  The  variable 
HC  is  scaled  according  to  the  following  table: 

TABLE  3.2-2.  HARDWARE  COMPLEXITY  (HC)  SCALING 


System  Class 


Typical  Range  in 
Part  Count 


Scale 


<  15,000 


1 


15,000  -  25,000 


2 


Equipment  or  Small  System 

Large  Equipment  or  Medium 
Sized  Equipment 


>  25,000 


«* 

I 

I 

» 

1 


Average  Number  of  Units  on  Test  Per  Month  (ANU).  AN U  is  determined  by  an 
estimate  of  the  average  number  of  systems  (i.e.,  "systems"  represented  by  the  value  for 
HC)  on  which  FRACAS  data  is  being  accumulated  on  a  monthly  basis. 


3.2.2  Models.  The  general  model  for  estimating  the  labor  for  implementing 
FRACAS/FRB  has  the  following  form: 


(3. 2. 2-1)  T104/5 


=(HC)C1  (DOI)C2  (ANU)C3 


where: 


104/5 


=  total  labor  (hours) 


=  hardware  complexity  (see  3.2.1) 


duration  of  FRACAS/FRB  implementation  (months) 


=  average  number  of  units  (see  3.2.1) 


C},  C-2,  C3  =  model  coefficients  (see  Table  3. 2. 2-1) 


Equation  (3. 2. 2-1)  covers  FRACAS/FRB  activities  occurring  during  all  development 
phases  of  the  program:  manufacturing,  equipment  checkout  and  test,  and  during  system 
integration,  checkout  and  test  (both  in-plant  and  on-site).  The  labor  involved  in 
conducting  detailed  laboratory  failure  analyses  (e.g.,  physics  of  failure)  is  not  included 
in  (3. 2. 2-1).  This  labor  can  run  from  20  to  100  hours  or  more  per  analysis  depending  on 
the  depth  of  analysis.  In  actual  practice,  the  number  of  these  analyses  depends  not  only 
on  the  complexity  of  the  system  but  also  on  the  maturity  of  the  design,  development 
phase  and  the  planned-versus-actual  growth  in  RAM.  Similarly,  the  time  spent  by  the 
program  Technical  Director,  Quality  Assurance  and  the  engineering  design  specialties  in 
participating  in  FRB  activities  and  in  developing  engineering  design  fixes  is  not  included 
in  (3. 1.2-1).  However,  reliability  growth  monitoring  and  assessment  (MIL-S'l'D-785  Task 
302)  activities  of  the  systems  in  the  data  base  were  logged  under  the  FRACAS/FRB 
task  and  therefore  included  in  (3. 1.2-1). 


Table  3.2. 2-1  provides  two  models  for  estimating  task  labor  based  on  available 
data.  For  example,  if  only  the  hardware  complexity  and  the  duration  of  FRACAS 
implementation  are  known  (e.g.,  HC=2  and  D01=24  months)  then  reference  to  Table 
3. 2. 2-1  results  in  the  following  estimate: 


TJ  04/5  =  (HC)0-251  (DOI)  2,496 


=  (2)0*251  (24)2'496 


=  3316  hours  (=  138  hours/month) 


till 


& 

& 


iVirNl  k*  A  I 


I  ABLE  3. 2. 2-1.  CEK  MODELS  FOR  FRACAS/FRB 


Equation 

No. 

Model  Coefficients  (Cj) 

HC 

DOI 

ANU 

P(  F) 

R2 

1 

0.251 

2.496 

- 

0.0001 

0.994 

2 

0.81  0 

2.279 

0.221 

0.0001 

0.995 

It  should  be  noted  that  Equation  No.  1  implicitly  assumes  an  "average"  number  of  units 
of  complexity  lit;  (i.e.,  the  data  base  average  for  ANU)  is  in  test  for  duration  DOI.  In 
this  sense,  the  coefficients  of  the  equation  are  adjusted  to  account  for  the  absence  of 
ANU. 

l  he  last  two  columns  in  fable  3.2.2— 1  provide  the  model  fit  statistics.  Both 
equations  are  considered  excellent  fits,  with  significance  tests  exhibiting  probabilities 
well  below  0.05  and  R  2  values  well  in  excess  of  0.9.  Appendix  B  provides  the  complete 
regression  analysis  results  for  these  four  equations. 

3.2.3  Cost  Tables.  Equation  No.  2  of  Table  3.2. 2-1  was  used  to  generate 
Table  3. 2. 3-1.  The  variable  representing  hardware  complexity  (HC)  defined  above,  can 
also  be  interpreted  in  terms  of  system  MTBF.  Table  3. 2. 3-2  categorizes  HC  by  major 
equipment/system  and  associates  part  count  with  MTBF  ranges. 

TABLE  3. 2. 3-1.  TASKS  104  AND  105  FRACAS/FRB  (TOTAL  HOURS) 


Duration  of  Implementation 


12  Months 

24  Months 

36  Months 

Average 
Number 
of  Units 

Equip/ 

Small 

System 

Large 

Equip/ 

System 

Large 

System 

Equip/ 

Small 

System 

Large 

Equip/ 

System 

Large 

System 

Equip/ 

Small 

System 

Large 

Equip/ 

System 

Large 

System 

1 

288. 

505. 

701. 

1397. 

2449. 

3401. 

3520. 

6170. 

8568. 

10 

479. 

840. 

1167. 

2326. 

4077. 

5661. 

5859. 

10270. 

14262. 

20 

559. 

979. 

1360. 

2711. 

4572. 

6600. 

6830. 

11973. 

16626. 

30 

611. 

1071. 

1488. 

2966. 

5199. 

7219. 

7471. 

13097.  | 

_ i 

18187. 

TABLE  3. 2. 3-2.  HARDWARE  COMPLEXITY  VERSUS  MTBF 


Category 


Equipment  or  Small  System 


Average  Number 
of  Electronic  Parts 


Range  of  MTBF  (Hours)* 


Large  Equipment  or  System 


7,500 

20,000 

50,000 


2,500  -  13,000 
100  -  500 


$1 


a 


A  medium  sized  system,  for  example,  consisting  of  20,000  electronic  parts  would 
be  expected  to  have  a  range  in  series  MTBF  of  between  100  and  500  hours.  If  an 
average  of  10  system?  are  reported-on  during  a  24-month  FRACAS  period,  Table  3. 2.3-1 
estimates  the  labor  at  4077  hours.  This  labor  does  not  include  laboratory  failure 
analysis,  which  can  be  added  by  using  Table  3.2.3- 2  and  assuming  a  fraction  of  the 
expected  number  of  failures  will  require  analysis.  This  can  be  accomplished  as  follows: 


TpA  =  (Expected  failures/mo)(Fraction  requiring  failure  analysis) 
(Labor  hours/analvsis) 


where:  (Expected  failures/mo)  =  (MTBF)'l(Operating  hr s/ mo/svstem) 

(10  systems) 

=  (300)'1 2  (720)(10)  =  24'  ^ 


Therefore,  in  this  example: 


(Fraction  requiring  failure  analysis)  =  1.0^) 


(Labor  hours/analysis)  =  60  (using  the  mid-point  in  the  range 

discussed  in  3.2. 2) 


Tfa  =  (24)(1.0)(60)  =  1440  hours/mo  for  failure  analysis, 


and  the  total  24  month  FRACAS/FRB  effort  is: 


T1  04/5  =  4077  +  (24X1440)  =  38,637  hours 


3.3  Modeling/ Allocations  (Tasks  201  and  202).  The  extent  of  activities  involved  in 
modeling  a  system  architecture  and  the  subsequent  allocation  of  requirements,  using 
this  model,  to  a  prescribed  set  of  equipment  and  preliminary  R&M  prediction  data  are 
directly  related  to:  (1)  the  number  of  successful  operating  modes,  (2)  the  complexity  of 
the  R <5c IV)  requirements,  and  (3)  the  number  of  equipment  types.  For  the  systems  in  the 
data  base,  standard  series-paralleled  configurations  were  modeled  using  generalized 
computer  models.  Therefore,  modeling  and  allocation  activities  would  be  somewhat 
higher  without  these  computer  aids.  For  more  complex  models  (see  below),  the 
development  of  unique  analytical  models  or  simulation  programs  were  often  required 
which  resulted  in  a  higher  expenditure  of  labor.  Similarly,  repairable  systems  have  an 
added  degree  of  complexity  because  of  the  added  requirements  associated  with 
maintainability  which  affects  the  allocation  effort  as  well  as  the  modeling  activities. 


3.3.1  Input  Data  Requirements.  Table  3.3-1  gives  the  independent  variables  having 
significant  correlation  with  the  modeling  and  allocation  activities  and  their  ranges  of 
applicability.  The  following  paragraphs  define  the  variables  and  identify  any 
constraints  on  their  use  in  the  CER  models  (3.3.2): 


Modeling/ Allocation  Complexity  (MAC).  MAC  consists  of  three  levels  of 
complexity:  (1)  minimal  complexity  (MAC=1)  representing  a  series  configuration  or  a 


(1)  Using  the  average  MTBF  for  20,000  parts. 

(2)  Assume,  for  example,  that  for  every  random  failure  (determi 
lab  analysis. 


ined  by  MTBF)  requires 


TABLE  3.3-1.  PROGRAM  FACTORS  CORRELATED  WITH 
MODE  LING/ ALLOCATION  ACTIVITIES 


r 

t 


Range  of  Application 


Minimum 

Maximum 

Uni 

1 

3 

Sea 

7 

445 

- 

1 

4 

- 

Factor 

Modeling/ Allocation  Complexity  (MAC) 
Number  of  Unique  Items  (NOU) 
Requirements  Complexity  (RC) 


very  small  amount  of  redundancy;  (2)  medium  complexity  (MAC=2)  involving  a  simple 
redundant  system  (i.e.,  a  series-parallel  network  without  interdependencies  which  can 
be  represented  by  a  general  computer  model);  and  (3)  maximum  complexity  (MAC=3) 
involving  any  combination  of  nested  structures,  dependent  subsystems,  path  sharing, 
etc.  which  require  extensive  model  development  effort. 

Number  of  Unique  Items  (NOU).  This  variable  applies  to  the  number  of  unique 
items  involved  in  the  allocation  process.  For  the  systems  in  the  data  base,  an  "item"  is 
generally  defined  as  a  procurable  unit  where  the  allocated  R  and/or  M  values  become 
procurement  specifications. 

Requirements  Complexity  (RC).  This  variable  pertains  to  the  number  of  distinct 
models  of  the  complexity  specified  by  MAC  for  a  complex  system.  For  example,  an  air 
defense  system  may  consist  of  several  segments  each  of  which  has  its  own  unique 
functional  configuration  and  corresponding  set  of  R/M  requirements.  RC  acts  as  a 
significant  multiplying  factor  to  modeling  and  allocation  complexity  (MAC). 

3.3.2  Models.  The  general  model  for  estimating  the  labor  requirements  for  system 
modeling  and  allocation  activities  has  the  following  form: 


where: 


(3.3. 2-1)  T201/2 
e:  t201/2 


=  (MAC)C1  (NOU)c2  (RC)C3 
=  total  labor  (hours) 


MAC  =  modeling/allocation  complexity  (see  3.3.1) 

NOU  =  number  of  unique  items  (see  3.3.1) 

RC  =  requirements  complexity  (see  3.3.1) 

Cj,  C2,  C3  =  model  coefficients  (see  Table  3. 3. 2-1) 


System  modeling  activities  consist  of  the  development  of  reliability  block  diagrams  and 
equations  for  estimating  the  various  systems  R<5fM  and  availability  parameters.  More 
complicated  systems  in  the  data  base  required  more  extensive  requirements  analysis 
and  a  top-level  "functional  FMEA"  (i.e.,  to  determine  the  effects  of  reconfiguration 
time,  switching  time,  fault  detection  etc.)  as  a  basis  for  developing  the  estimating 
equations  (or  simulation  program).  This  "FMEA"  effort  is  only  for  supporting  the  model 
development.  Contractually  required  FMECA's  are  included  under  Task  204  (see  3.5). 
After  the  initial  development,  the  models  and  equations  are  updated  as  the  system 
design  evolves  into  its  final  form,  usually  prior  to  the  program  critical  design  review 
(CDR). 

RAM  allocation  activities  involve  use  of  the  above  models  and  inputs  from  the 
R&M  predictions  (Task  203  -  see  3.4)  to  flow  down  the  system  requirements  to 
designers,  subcontractors  and  vendor  product  specifications.  These  activities  include 
interfacing  with  and  providing  detailed  R&M  inputs  (e.g.,  MTBF’s,  MTTR's,  M|y]ax's, 
fault  detection  and  fault  isolation  criterion)  to  design  and  logistics  analysis 
organizations. 

Table  3. 3. 2-1  provides  two  models  for  estimating  the  combined  activities  of 
modeling  and  allocation.  For  a  system  composed  of  three  complex  subsystems  (i.e., 
\1AC=3  for  each  subsystem  which  has  its  own  set  of  requirements)  and  consistency  of  20 
unique  items  (system  total),  Equation  No.  2  gives  the  following  labor  estimate: 

1201/2  =  (MAC)2-03l(RC)2-07l(NOU)°-798 
=  (3)2.031(3)2.071(20)0.798 

=  989  hours 

It  should  be  emphasized  in  this  example  that  the  three  subsystems  are  interpreted  to  be 
uniquely  complex  each  requiring  some  model  development. 


TABLE  3. 3. 2-1.  CER  MODELS  FOR  MODELING  AND  ALLOCATIONS 


Equation 

No. 

Model  Coefficients  (Cj) 

MAC 

RC 

NOU 

P(>F) 

R2 

1 

4.350 

- 

0.866 

0.0003 

0.935 

2 

2.031 

2.071 

0.798 

0.001  1 

0.950 

The  last  two  columns  in  Table  3. 3. 2-1  provide  the  model  fit  statistics.  Both 
equations  are  considered  excellent  fits,  with  significance  tests  with  probabilities  well 
below  0.05  and  R2  values  in  excess  of  0.9.  Appendix  B  provides  the  complete  regression 
analysis  results  for  these  equations. 


3.3.3  Cost  Tables.  Table  3. 3. 3-1  represents  a  simplified  form  of  equation  No.  2 
where  RC  is  limited  to  four  distinct  modeling  activities. 
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TABLE  3. 3. 3-1.  TASKS  201  AN L)  202  MODELING/ ALLOCATIONS  (HOURS) 


Unique 

Items 

Distinct  Models  at 

M  AC  =  1 

Distinct  Models  at 
MAC  =  2 

Distinct  Models  at 

MAC  =  3 

1 

2 

fl 

B 

fl 

2 

B 

B 

B 

2 

B 

4 

25. 

13. 

55. 

127. 

231. 

53. 

519. 

942. 

122. 

51 1. 

1183. 

2147. 

23. 

95. 

221. 

401. 

93. 

903. 

1639. 

211. 

888. 

2058. 

3734. 

69. 

289. 

668. 

1213. 

281. 

1179. 

2731. 

4956. 

639. 

2687. 

6222. 

11291. 

95. 

399. 

924. 

1676. 

388. 

3775. 

6849. 

884. 

3713. 

8600. 

15605. 

119. 

502. 

1162. 

2109. 

488. 

4749. 

8617. 

1112. 

4672. 

10820. 

19633. 

3.4  R&M  Predictions  (Task  203).  The  amount  of  effort  required  to  make  R&M 
predictions  depends  on  the  complexity  of  the  hardware  and  the  level  of  detail  at  which 
the  prediction  is  made.  Systems  in  the  data  base  for  which  detailed  predictions  were 
made  utilized  a  computerized  version  of  iVlIL-HDBK-217.  Therefore,  the  effort 
involving  tedious  hand  calculations  for  each  component  is  replaced  by  the  task  of  coding 
component  characteristics  for  computer  input. 

3.4.1  Input  Data  Requirements.  The  following  paragraphs  define  the  independent 
variables  having  correlation  with  the  prediction  effort  and  Table  3.4-1  gives  the 
application  ranges: 

Hardware  Complexity  (HC).  This  variable  has  the  same  meaning  as  in  3.2.1. 

Level  of  Detail  (LOP).  LOD  consist  of  three  levels:  (1)  minimum  level  (LOD=l)  in 
which  the  effort  involves  review  of  vendor  furnished  data;  (2)  medium  level  (LOD=2) 
requiring  circuit  card  assembly,  power  supply,  etc.  estimates  based  on  similar-to 
assessments  and  engineering  analysis;  and  (3)  maximum  level  (LOD=3)  requiring  a 
detailed  piecepart  prediction  per  MIL-HDBK-217  including  appropriate  thermal  and 
electrical  stress  analyses. 

Reporting  Formality  (RF).  R&M  prediction  results  can  either  be  formally 
submitted  (RF=2)  resulting  in  added  effort  in  subsequent  technical  clarifications  and 
discussions  with  the  customer,  or  they  can  be  informal  ( R  F=  1 )  subject  to  only  an 
internal  management  review.  In  the  formal  review,  the  added  effort  appears  to  be 
based  on  customer  unfamiliarity  with  the  prediction  presentation  format  and/or 
concerns  over  the  prediction  ground  rules  and  assumptions. 


TABLE  3.4-1.  PROGRAM  FACTORS  CORRELATED  WITH 
R&M  PREDICTION  ACTIVITIES 


Factor 

Range  of  Application* 

Minimum 

Maximum 

Units 

Hardware  Complexity  (HC) 

I 

3 

Scale 

Level  of  Detail  (LOD) 

1 

3 

Scale 

Reporting  Formality  (RF) 

1 

2 

Scale 

Percentage  of  Commercial  (POC) 

1 

4 

Scale 

Repairable/Non-Repairable  (RNR) 

1 

2 

Scale 

*  Additional  conditions  on  the  application  of  these  factors  are  required  (see  3.4.2) 


Percent  of  Commercial  (POC).  The  R<3rM  prediction  activity  is  affected  by  the 
percentage  of  commercial  (POC)  hardware  since  the  commercial  hardware  predictions 
(at  the  appropriate  level  of  detail)  are  generally  performed  by  the  commercial 
manufacturer  and  only  reviewed  by  the  contractor.  Although  the  manufacturer's  effort 
in  making  R&M  predictions  is  included  in  his  bid  to  the  contractor,  this  cost  could  not 
be  broken-out  and  included  in  the  CER  models  for  the  systems  in  the  data  base.  The 
value  of  POC  is  scaled  as  follows: 


Percentage  of  Commercial 

Hardware 

Scale 

0-25 

4 

26-  50 

3 

51  -  75 

2 

76  -  100 

1 

Note  that  the  highest  value  on  the  scale  (POC=4)  denotes  the  least  amount  of 
commercial  hardware. 


Repairable/Non-Repairable  (RNR).  This  variable  pertains  to  whether  or  not  the 
system  is  repairable  (RNR=2)  while  performing  its  basic  mission  with  only  temporary 
disruption  of  service.  For  example,  ground  based  systems  such  as  air  defense  systems 
are  repairable,  but  fighter  aircraft  electronic  systems  are  not  repairable  (RNR  =  1)  while 
on  a  mission.  Systems  that  are  repairable  in  this  sense  generally  require  maintainability 
prediction  effort. 

3.4.2  Models.  The  general  model  for  estimating  the  labor  involved  in  making  R<5rM 
predictions  has  the  following  form: 


(3. 4.2-1) 

T203 

C  C  G,  C  C 

=  (HC)  ^LOD)  2  (RF)  (POC)  4  (RNR)  5 

where: 

T 

*203 

=  task  labor  (hours) 

HC 

=  hardware  complexity  (see  3.2.1) 

LOD 

=  level  of  detail  (see  3.4.1) 

RF 

=  reporting  formality  (see  3.4.1) 

POC 

=  percent  of  commercial  hardware  (see  3.4.1) 

R  NR 

=  repairable/non-repairable  index  (see  3.3.1) 

cl. 

C2’  C3’  C4’  C5 

=  model  coefficients  (see  Table  3. 4.2-1) 

Equation  (3.4.2-1)  covers  both  R  and  M  prediction  effort  starting  with  the  earliest 
preliminary  predictions  and  including  all  subsequent  updates.  For  a  minimal  level  of 
detail,  this  would  involve,  for  example,  reviewing  and  implementing  changes  (updates) 
to  vendor-furnished  data.  For  a  maximum  level  of  detail,  the  effort  may  start  with 
similar-to  estimates  at  this  circuit  card  or  unit  level  with  updates  eventually  based  on 
detailed  MIL-HDBK-217  and/or  MIL-HDBK-472  predictions  as  the  design  implementa¬ 
tion  details  become  known.  Most  of  the  systems  in  this  data  base  required  a  combina¬ 
tion  of  reviewing  vendor  R<5cM  data  and  MIL-HDBK-217  predictions  for  newly  developed 
equipment  and/or  interfaces.  In  these  cases,  an  assessment  was  made  based  on 
interviews  with  the  responsible  R<5cM  engineers  as  to  where  most  of  this  labor  was 
expended  and  a  level  of  detail  index  was  assigned  accordingly. 

When  MIL-HDBK-217  predictions  were  required  (i.e.,  for  maximum  level  of  detail), 
the  systems  in  the  data  base  generally  included  effort  for  the  analysis  of  newly  designed 
circuits  to  determine  part  operating  stresses  (electrical  and  thermal)  as  part  of  the 
prediction  effort.  Aside  from  the  task-  time  analyses,  MIL-HDBK-472  prediction  efforts 
also  included  analyses  of  equipment  fault  isolation  capability  and  the  derivations  of 
inputs  for  logistics  support  analyses  (LSA's)  such  as  MTTR's,  MMAX's,  fault  isolation 
ambiguities  etc..  These  efforts  are  classified  separately  (Tasks  205  -  maintainability 
analysis  and  207  -  preparation  of  LSA  inputs)  in  MIL-STD-470,  but  were  too  integral  to 
the  prediction  task  effort  to  be  broken  out  separately. 

Table  3. 4. 2-1  provides  four  models  for  estimating  R<5cM  prediction  task  labor  based 


on  available  data.  The  scales  on  the  factors  given  in  Table  3.4-1  cannot  be  used 
independently  since  the  data  base  does  not  support  extreme  combinations  (e.g.,  all 
factors  taken  at  their  minimum  values).  The  following  conditions  apply  to  the  models  in 
Table  3. 4. 2-1: 

Equation  No.  1:  HC  +  LOD  +  RF  >_  4 
Equation  No.  2:  HC  +  LOD  +  RF  +  POC  >  8 


Equation  No.  3:  HC  +  LuL)  +  POC  +  RNR  _>  7 

Equation  No.  4:  HC  +  LOD  +  RF  +  POC  +  RNR  >.9 

If  the  input  data  does  not  satisfy  the  corresponding  condition  given  above  for  a  selected 
equation,  the  computed  result  will  be  invalid.  For  example,  the  effort  required  for  a 
detailed  R3cM  Prediction  of  a  complex  (i.e,  HC  =  3)  repairable  system  consisting  of  50 
percent  commercial  equipment  satisfies  the  condition  for  Equation  No.  3  since: 

HC  +  LOD  +  POC  +  RNR  =  3+  3  +  3  +  2  =  ll>7 

Therefore,  the  prediction  labor  is: 

T-203  =  (HC)°*786(LOD)2*103(POC)l-643(RNR)2-076 
=  (3)0-786(3)2.103(3)l. 643(2)2.076 

=  613  hours 


TABLE  3.4. 2-1.  CER  MODELS  FOR  R&M  PREDICTIONS 


Equation 

No. 

Model  Coefficients  (Ci) 

HC 

LOD 

RF 

POC 

RNR 

P(  >  F ) 

R  2 

1 

1.125 

3.285 

5.164 

- 

- 

0.0001 

0.947 

2 

1.308 

2.469 

3.343 

1.489 

0.0001 

0.963 

3 

0.786 

2.103 

- 

1.643 

2.076 

0.0001 

4 

0.861 

2.139 

1.840 

1.315  j 

1.438 

0.0001 

0.971 

The  last  two  columns  in  Table  3.4.2-1  provide  the  model  fit  statistics.  The  equations  in 
the  table  are  all  considered  good  fits,  with  significance  tests  exhibiting  probabilities 
well  below  0.05  and  R2  values  in  excess  of  0.9.  The  complete  regression  analysis  results 
for  these  equations  are  given  in  Appendix  B. 

3.4.3  Cost  Tables.  Equation  No.  4  of  Table  3. 4. 2-1  was  used  to  generate  cost 
fables  3.4. 3-1  and  3.4. 3-2  for  non-repairable  and  repairable  equipment/systems.  As  in 
previous  cost  tabulations,  hardware  complexity  (HC)  is  categorized  into  three  groupings 
of  equipment/systems  (also  see  Table  3. 2. 3-2). 

Using  the  above  example,  Table  3. 4. 3-2  would  be  used  since  the  system  is 
repairable.  The  level  of  the  prediction  is  "detailed"  (i.e.,  part  level),  the  system  is 
complex  (i.e.,  a  large  system),  and  consists  of  50  percent  commercial  equipment.  The 
last  column  of  Table  3.4.3-2  shows  two  cases:  informal  reporting  (RF=0)  resulting  in 
313  hours,  and  formal  reporting  (RF=1)  resulting  in  1122  hours. 


TABLE  3.4. 3-2.  TASK  203  PREDICTIONS  -  REPAIRABLE  (HOURS) 


Prediction  Level  of  Detail 


Equipment 

Item  Analyzed 

Assembly 

Hem  Analyzed 

Part 

Item  Analyzed 

Percentage 

Commercial 

Equip/ 

Small 

System 

Large 

Equip/ 

System 

Large 

System 

Equip/ 

Small 

System 

Large 

Equip/ 

System 

Large 

System 

Equip/ 

Small 

System 

Large 

Equip/ 

System 

Large 

System 

f  76-100 

* 

• 

* 

* 

* 

* 

4c 

73. 

KF  =  0  < 

1  51-75 

* 

* 

* 

i 

* 

77. 

♦ 

129. 

183. 

|  26-50 

* 

* 

30. 

< 

93. 

132. 

122. 

221. 

313. 

k  0-25 

* 

31. 

44. 

75. 

136. 

193. 

178. 

324. 

459. 

f  76-100 

* 

* 

* 

4 

* 

110. 

4c 

185. 

262. 

RF  =  1  < 

i  51-75 

* 

* 

63. 

4 

194. 

275. 

255. 

462. 

656. 

|  26-50 

* 

76. 

107. 

183. 

333. 

471. 

436. 

792. 

1122. 

1 

^  0-25 

61. 

111. 

157. 

268. 

487. 

690. 

638. 

1159. 

1643. 

♦The  models  do  not  apply  for  these  conditions. 


3.5  FMECA  (Task  204).  The  amount  of  effort  required  to  conduct  a  failure  mode, 
effects  and  criticality  analysis  depends  on  the  number  of  items  involved  in  the  analysis, 
how  complex  the  system  is  and  depth  of  the  analysis  (or  level  of  detail).  All  of  the 
analyses  for  systems  included  in  the  data  base  were  conducted  in  accordance  with 
MIL-STD-1 629. 

3.5.1  Input  Data  Requirements.  The  following  paragraphs  define  the  independent 
variables  having  correlation  with  FMECA  effort  and  Table  3.5-1  gives  the  application 
ranges: 


Number  of  Unique  Items  (NOU).  This  variable  requires  an  estimate  of  the  total 
number  items  (as  defined  below)  requiring  an  FMECA  at  the  specified  level  of  detail. 


Level  of  Detail  (LOD) 
Equipment  Interface 
Equipment 

Circuit  Card  Assembly 
Piecepart 


Type  of  Items  Counted 
Equipment 
Equipment 
Circuit  Card 
Circuit  Card 


For  example,  if  two  equipment  items  required  an  FMEA  at  the  equipment  level 
(i.e.,  considering  the  equipment  as  "black  box"),  and  five  newly  designed  cards  (used  in 
both  equipment)  required  an  FMEA  at  the  piecepart  level,  then  the  value  of  NOU  for 
the  equipment  level  would  be  2  and  the  value  of  NOU  for  the  piecepart  level  would  be 
5.  Each  FMEA  would  be  computed  separately  using  a  model  defined  in  3.5.2  and  then 
combined  to  obtain  the  total  FMEA  effort  estimate. 

Hardware  Complexity  (HC)  -  This  variable  has  the  same  meaning  as  in  3.2.1. 

Level  of  Detail  (LOD)  -  LOD  consists  of  four  levels:  (LOD=l)  equipment  interface 
(i.e.,  the  effects  of  each  loss  of  function  on  the  system  is  examined); 

(LOD=2)  equipment;  (LOD=3)  circuit  card  assembly;  and  (LOD=4)  piecepart. 


TABLE  3.5-1.  PROGRAM  FACTORS  CORRELATED  WITH  FMECA  ACTIVITIES 


Factor 

Range  of  Application 

Minimum 

Maximum 

Units 

Number  of  Unique  Items  (NOU) 

3 

206 

- 

Hardware  Complexity  (HC) 

1 

3 

Scale 

Level  of  Detail  (LOD) 

1 

4 

Scale 

3.5.2  Models.  The  general  model  for  estimating  the  labor  required  to  conduct  an 
FMECA  has  the  following  form: 


(3. 5.2-1)  T 


where: 


204 


1 204 
NOU 

HC 

LOD 


=  (NOU)Cl  (HC)C2  (LOD)C3 
=  task  labor  (hours) 

=  number  of  units  analyzed  (see  3.5.1) 
=  hardware  complexity  (see  3.2.1) 

=  level  of  detail  (see  3.5.1) 


C,,  C9,  C.j  =  model  coefficients  (see  Table  3. 5. 2-1) 
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labor  for  conducting  a  "manual"  type  of  FMECA.  The 
automated  tools  to  assist  in  the  analysis  bookkeeping  and  report  generation  currently 
available  (e.g.,  R  \ L » t ' -  TK-84-244)  did  not  exist  for  the  systems  in  the  data  base. 
Automated  tools  can  greatly  reduce  the  labor  for  a  piecepart  FMECA  and,  to  a  lesser 
extent,  a  card-level  FMECA  (i.e.,  treating  the  circuit  card  as  a  black  box).  On  an 
equipment-level  or  equipment  interface  level  FMECA,  however,  the  effect  w  old  be 
insignificant  because  of  the  significantly  fewer  failure  modes  to  consider.  Therefore, 
the  FMECA  labor  estimated  by  equation  (3. 5. 2-1)  for  the  maximum  level  of  detail  will 
be  biased  on  the  high  side  whenever  automated  tools  are  employed.  Unlike  the  RAM 
prediction  task  discussed  previously,  combined  levels  of  FvlECA's  were  not  conducted 
for  any  of  the  systems  in  the  data  base.  For  complex  systems,  however,  it  is  reasonable 
for  an  FMECA  to  oe  required  on  newly  developed  equipment  and  also  on  the  total 
system  (e.g.,  an  equipment  interface  level  FMECA).  In  this  situation,  equation  (3. 5. 2-1) 
could  be  used  to  estimate  these  FMECA  efforts  separately  (see  below). 

Table  3. 5. 2-1  provides  three  models  for  estimating  FMECA  task  labor  based  on 
available  data.  Using  equation  No.  3,  suppose  that  a  detailed  (i.e.,  piecepart)  FMECA  is 
to  be  conducted  on  five  interface  cards  that  are  new  design  and  on  the  entire  system 
consisting  of  15  major  equipment  (i.e.,  conducted  at  the  equipment  interfaces.)  The 
total  system  is  considered  complex  (i.e.,  HC  =  3).  The  equation  variables  and  FMECA 
estimates  are  given  below: 

a.  C  i  rc  u  i  t  (3  a  rd  F  M  E  C  A : 

T204  =  (NOU)1-362  (HC)1-897  (LOD)1-315 
=  (5)1.362  (3)1.897  (4)1.315 

=  445  hours 

b.  System  FMECA: 

^204  =  (lo)1-362  (3)1-897  (1)1*31  5 
=321  hours 


The  total  FMECA  effort  from  a.  and  b.  is  766  hours. 


Equation 

No. 


TABLE  3.5. 

2-1.  CEH  MODELS  FOR  FMECA 

:  Model  Coefficients  (Ci) 

NOU 

HC 

LOD 

P(>F) 

R2 

1.899 

- 

- 

0.000  2 

0.945 

1.490 

2.666 

- 

0.0004 

0.981 

0.00  29 


CjU 

i 


The  last  two  columns  in  Table  3. 5. 2-1  provide  the  model  fit  statistics.  The 
equations  in  the  table  are  all  considered  good  fits,  with  significance  tests  showing 
probabilities  less  than  0.003  and  R^  values  greater  than  0.94.  The  complete  regression 
analysis  results  for  these  equations  are  given  in  Appendix  B. 

3.5.3  Cost  Tables.  Equation  No.  3  of  Table  3. 5. 2-1  was  used  to  generate  Table 
3. 5. 3-1,  approximations  of  the  previous  results  using  (3. 5. 2-1)  can  be  read  directly  from 
the  table  entering  the  appropriate  values  for  HC,  NOU  and  LOD. 

TABLE  3. 5. 3-1.  TASK  204  FMECA/FMEA  (HOURS) 


Level  of  Detail  (LOD) 


Equipment 

Interface 


Number 

of 

Unique 

Items 


Equipment 


Circuit  Card 
Assembly 


Piecepart 


Equip/  Large  Equip/  La 

|  Small  Equip/  Small  Eq 

(NOU)  System  System  System  Sys 


Equip/ 

Small 

System 

Large 

Equip/ 

System 

Equip/ 

Small 

System 

38. 

141. 

55. 

340. 

1265. 

496. 

872. 

3249. 

1274. 

2242. 

8349. 

3273. 

5760. 

21454. 

8410. 

3379.  12586. 


3.6  Reliability  Testing  (Tasks  301,  303  and  304).  The  amount  of  effort  required  to 
conduct  reliability-type  tests  depends  on  hardware  complexity,  the  number  of  pre-test 
environmental  screens  and  the  type  of  testing  (i.e.,  whether  testing  is  for  qualification 
(RQT),  production  acceptance  (PRAT)  or  both).  The  reliability  effort  involved  for  the 
systems  in  the  data  base  did  not  include  effort  required  to  maintain  the  test  facilities 
or  the  equipment  under  test.  Therefore,  the  total  effort  required  to  conduct  the  test 
would  be  considerably  higher  than  the  values  predicted  by  the  CER's  described  in  this 
section. 

3.6.1  Data  Input  Requirements.  The  following  paragraphs  define  the  independent 
variables  having  correlation  with  the  reliability  testing  activities.  Table  3.6-1  gives  the 
application  ranges: 

Hardware  Complexity  (HC).  This  variable  has  the  same  meaning  as  in  3.2.1. 

Reliability  Qualification  Testing  (RQT).  RQT  is  an  indicator  variable 
which  determines  whether  or  not  reliability  qualification  testing  is  conducted 
(RQT= 1  denotes  that  the  test  is  conducted,  RQT=0  denotes  that  the  test  is  not 
conducted) . 
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Production  Reliabilitv  Acceptance  Testing  (PRAT).  PRAT  is  an  indieatoj 


variable  which  determines  whether  or  not  production  reliability  acceptance 
testing  is  conducted  (PRAT-1  denotes  that  the  test  is  conducted,  PRAT=P 
denotes  that  the  test  is  not  conducted) . 

Number  of  Screens  (NOS).  NOS  represents  the  number  of  environmental  stress 
screens  (ESS)  that  are  conducted  prior  to  the  reliability  demonstration  test.  Individual 
screens  (e.g.,  temperature  cycling,  burn-in  etc.)  at  each  level  of  manufacture  (circuit 
card,  unit  equipment  and/or  system)  should  be  enumerated. 


TABLE  3.6-1.  PROGRAM  FACTORS  CORRELATE D  WITH  R-TEST  ACTIVITIES 


Hardware  Complexity  (HC) 

Reliability  Qualification 
Testing  (RQT) 

Production  Reliability 
Acceptance  Testing  (PRAT) 

Number  of  Screens  (NOS) 


Range  of  Application 


Minimum 


Maximum 


Indicator 


Indicator 


3.6.2  Models.  The  general  model  for  estimating  reliability  testing  labor  has  the 
following  general  form: 


(3. 6.2-1)  T 


where: 


301/3/4 


301/3/4 


=  C  j(HC)  +  c2(rqt)  +  c3(prat) 

+  c4(nos) 

=  task  labor  (hours) 

=  hardware  complexity  (see  3.2.1) 

=  number  of  pre-test  screens  (see  3.6.1) 
=  indicator  for  reliability 
qualification  testing  (see  3.6.1) 


PRAT 


indicator  for  production  reliability 


acceptance  testing  (see  3.6.1) 


'  I  ,c  „,crC4  =  model  coefficients  (see  Table  3.6.2- 1) 


Equation  (3.6. 2-1)  estimates  the  reliability  engineering  effort  for  reliability-type 
testing,  whether  the  test  is  qualification  (RQT)  or  production  reliability  acceptance 
(PH AT),  with  appropriate  pre-test  screens  (i.e.,  environmental  stress  screening  (ESS)). 
Labor  accounting  for  reliability  development/growth  was  considered  part  of  FRACAS 
(see  3.2)  for  the  system  in  this  data  base.  The  effort  includes  the  development  of  test 
plans  and  procedures,  monitoring  and  recording  the  test  activity  failure  classification 
and  resolution  of  hardware  failures  and  the  formal  documentation  of  the  test  results.  In 
this  sense,  the  reliability  engineering  effort  is  related  to  the  failure  activity  rather  than 
the  test  length.  During  ESS,  the  activity  is  much  the  same  as  the  FRACAS  effort, 
depending  on  the  number  of  tests  and  the  hardware  complexity.  Equation  (3.6.2— 1)  does 
not  include  the  test  engineering  effort  to  install  the  equipment/system  in  the  test 
environment,  develop  and  initiate  the  input  stimuli  and  generally  maintain  the 
equipment  and  test  facility.  The  effort  required  to  maintain  the  system  and  test 
facilities,  however,  does  relate  to  the  length  of  the  test,  and  can  typically  range  from  2 
to  4  or  5  manmonths  per  month  of  additional  effort  depending  on  the  system  complexity. 

Table  3. 6. 2-1  provides  four  models  for  estimating  the  reliability  engineering 
portion  of  the  test  effort. 


TABLE  3. 6. 2-1.  CER  MODELS  FOR  R-TESTING 


Equation 

No. 

Model  Coefficients  (Cj) 

P(  >  F) 

R2 

HC 

RQT 

PRAT 

NOS 

1 

285.1 

- 

- 

- 

0.0003 

2 

- 

444.1 

268.2 

- 

0.0014 

3 

- 

418.7 

239.4 

3.1 

0.0086 

0.931 

4 

126.9 

253.8 

173.0 

- 

0.0040 

Although  data  from  the  missing  variables  of  the  above  equations  (e.g.,  RQT,  PRAT  and 
NOS  are  missing  from  Equation  No.  1)  was  not  used  in  the  regression,  it  should  be  noted 
(as  in  previous  cases)  that  the  remaining  variables  are  constrained  by  the  data  base 
averages.  For  example,  suppose  the  test  system  will  go  through  5  temperature  cycling 
screens  (e.g.,  4  unit  level  and  1  system  level)  an  RQT  and  a  PRAT.  Equation  No.  3 
provides  the  following  estimate  of  the  reliability  engineering  effort: 

T30 1/3/4  =  418.7(RQT)  +  239.4(PRAT)  +  3.1(NOS) 


=  418.7(1)  +  239.4(1)  +  3.1(5)  =  673.6  hours 


Equation  No.  3  assumes  a  test  system  of  average  complexity  (i.e.,  weighted  according 
to  the  values  in  the  data  base).  On  the  other  hand,  if  the  only  information  known  is  that 
the  test  system  is  of  average  complexity  (i.e.,  HC=2)  then  Equation  No.  1  can  be  used 
to  estimate  the  effort  assuming  a  nominal  (i.e.,  the  data  base  average)  amount  of 
reliability  screening  and  testing: 

t30i/4=  285.1(HC)  =  285.1(2)  =  570.2  hours 

The  last  two  columns  in  Table  3. 6.2-1  provide  the  model  fit  statistics.  The  equations  in 
the  table  are  considered  good  fits,  with  significance  tests  showing  probabilities  less 
than  0.004  and  R^  values  greater  than  0.90.  The  complete  regression  analysis  results 
are  given  in  Appendix  B. 

3.6.3  Cost  Tables.  Equation  No.  3  of  Table  3. 6.2-1  was  used  to  generate  Table 
3. 6. 3-1.  Note  that  the  first  column  represents  the  labor  hours  for  reliability  support  of 
the  screening  tests  only  and  the  last  column  includes  pre-screening  effort  as  well  as  the 
reliability  support  for  RQT  and  PRAT. 


TABLE  3. 6.3-1.  TASKS  301,  303,  304  REL  TESTING  (HOURS) 


No.  of 
Screens 


Screening  Only 


Screening 
and  PRAT 


Screening 
and  RQT 


Screening,  R  QT 
and  PRAT 


4.0  INVESTIGATION  INTO  THE  FEASIBILITY  OF  DERIVING  COST/BENEFIT  RATIOS 


4.1  Candidate  Approaches.  There  are  several  possible  approaches  to  determining 
R &  VI  program  tasks  benefits.  Each  approach  must  assign  values  to  relevant  and 
measurable  figures  of  merit  such  as  MTBF,  MTTR,  BIT  effectiveness,  etc.  The  exact 
figure  of  merit  to  be  used  is  dependent  on  the  approach  found  feasible  and  the  degree  to 
which  information  sources  exist  which  support  evaluation.  Four  approaches  were 
investigated  and  one  of  these  was  selected  to  develop  a  preliminary  model. 

Ideally,  the  method  of  developing  R&M  Task  Benefit-Indices  is  to  make  a 
quantitative  assessment  of  the  benefit  prior  to  and  after  application  of  each  individual 
task.  The  R 3c  VI  task  benefit  would  then  be  the  gain  in  benefit  experienced  by  applying 
the  task.  Aside  from  the  availability  of  such  data,  this  method  is  over  simplified,  and 
suffers  from  many  inherent  difficulties.  For  example,  FMEAs  can  be  done  at  various 
system  and  subsystem  levels,  so  that  the  benefit  gained  from  performing  an  FiVIEA  is 
not  just  a  function  of  whether  or  not  the  FMEA  was  applied,  but  also  the  level  of 
application  (i.e.,  "how  much"  FMEA  is  applied).  Also,  while  many  R&M  tasks  are  only 
quantifiable  as  either  being  present  or  not,  the  benefit  gained  by  applying  such  a  task 
could  depend  on  other  factors  as  well,  including  equipment  complexity,  and  the 
presence  of  other  R&M  tasks.  A  reliability  modeling  effort,  for  example,  would  make 
very  little  sense  without  the  associated  prediction  effort  which  provides  the  input  data 
used  by  the  models.  Thus,  it  is  difficult  to  accurately  assess  the  benefit  gained  from 
each  task  separately  either  quantitatively  or  subjectively.  It  is  tacitly  assumed  in  the 
approaches  described  below,  therefore,  that  R&M  tasks  are  to  be  grouped  appropriately. 

4.1.1  "Task- By-Task"  Assessment  Based  on  Hardware  Engineering  Changes.  The 
reliability  and  maintainability  impact  of  the  hardware  changes  caused  by  the  individual 
task,  or  composite  set  of  tasks,  is  assessed  to  determine  an  appropriate  change  in  the 
figure  of  merit  associated  with  the  task(s).  This  method  is  used  for  each  task  or  group 
of  tasks  throughout  the  program  until  a  point  where  the  reliability  and  maintainability 
characteristics  of  the  equipment  have  been  reasonably  established,  usually  at  the 
conclusion  of  qualification  testing. 

The  primary  advantage  of  a  task-by-task  assessment  of  this  type  is  that  the 
approach  is  direct  and  traceable.  The  assessment  of  an  improvement  in  equipment 
reliability  or  maintainability  is  dependent  on  evidence  of  a  design  change  which  was 
caused  by  the  results  of  the  task.  The  assessment  of  changes  caused  by  a  task  requires 
that  detailed  engineering  records  be  available  throughout  the  development  program. 

This  level  of  detail  is  potentially  realizable  for  a  development  program  when  hardware 
changes  are  formally  documented  in  engineering  change  proposals  (ECPs)  and  in  test 
reports. 


I 


I 


The  main  disadvantage  found  with  a  direct  task-by-task  assessment  is  that  the 
amount  of  program  data  and  level  of  detail  required  to  use  the  method  is  prohibitive. 
Engineer  notebooks,  laboratory  record  books,  and  deliverable  data  items  are  normally 
available  from  the  early  equipment  development  period  but  these  records  do  not  contain 
sufficient  detail  to  allow  the  large  number  of  informal  changes  which  occur  as  a  result 
of  R&M  analyses  to  be  identified  or  evaluated.  Additionally,  the  direct  task  analysis 
approach,  which  necessarily  relies  on  engineering  judgment  for  assessment  of  changes 
due  to  R&M  task  performance  (as  opposed  to  other  reasons)  would  be  subjective  and 
unique  to  Hughes.  Therefore,  a  task-by-task  assessment  approach  is  not  recommended 
as  a  basis  for  developing  a  benefits  relationship. 


4.1.2  Task  Assessment  by  Case  Study.  Case  studies  which  investigate  the  benefit 
of  tasks  in  an  R&M  program  tend  to  be  subjective  but  are  useful  as  general  support  for 
more  quantitative  analyses.  The  Institute  for  Defense  Analysis  (IDA)  has  conducted  a 
number  of  these  studies  over  the  past  several  years.  Although  the  RJc.V]  benefits  for  a 
given  case  are  somewhat  unique  to  the  situation,  an  assessment  across  a  number  of 
programs  should  identify  the  major  benefiting  tasks.  The  following  paragraphs 
summarize  six  IDA  case  studies  which  had  applicable  R&M  information: 


F-15  AN/APG  63  RADAR.  This  case  study  [3]  also  found  that  strong  customer  and 
program  office  management  support  are  necessary  for  an  effective  R&M  program.  The 
study  questioned  the  value  of  the  "Laboratory"  R  Demo  and  found  that  "failure-free" 
burn-in/aeceptance  testing  to  be  effective.  Further,  a  field-FRACAS  program  was 
judged  to  be  the  most  effective  tool  for  identifying  and  correcting  reliability  problems. 


F/A-18,  AN/APG-65  RADAR.  This  study  [4]  did  an  item  by  item  qualitative/ 
quantitative  analysis  of  the  R&M  Tasks  performed  during  this  program.  The  tasks 
which  provided  the  "most  return  for  the  money"  were  TAAF,  FRACAS,  FKB  and  the 
worst  case/stress  analyses.  The  study  was  critical  of  the  100%  piece  part  level  FiVlECA 
as  not  effective.  However,  the  study  did  find  value  in  the  FMECA  as  a  tool  at  other 
levels. 


F—  1 6  AN/APG-66  RADAR.  This  case  study  [5]  pointed  out  that  strong  management 


support  from  both  the  customer  and  program  office  are  necessary  for  an  effective  R&.Vl 
program.  The  "lessons  learned"  found  that  ESS  at  all  levels  of  test  to  be  very 
effective.  As  in  the  previous  case,  TAAF,  FRACAS  and  FRB  were  again  identified  as 
effective. 


T700  Jet  Turbine  Engine.  The  primary  factor  this  study  [6]  pointed  out  was  that 
reliability  and  maintainability  must  be  "designed-in".  The  lessons  learned  dealt  with 
quality  factors  more  than  purely  R  or  M  factors.  However,  vendor  control  was  deemed 
to  be  very  necessary. 


AN/APN-128  Lightweight  Doppler  Navigation  System.  This  case  study  [7]  also 
stressed  the  need  for  management  involvement  for  an  R<5cM  program  to  be  successful. 
The  R  v5c  iVl  tasks  found  to  be  effective  were  the  design  analyses  such  as  worst  case,  and 
thermal  stress  (hot  and  cold)  analyses.  ESS  testing  at  all  levels  and  FRACAS  was  also 
considered  valuable. 
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4.0  INVESTIGATION  IN  TO  THE  FEASIBILITY  OF  DERIVING  COST/ BE NE FIT  RATIOS 


-1. 1  Candidate  Approaches.  There  are  several  possible  approaches  to  determining 
RAM  program  tasks  benefits.  Each  approach  must  assign  values  to  relevant  and 
measurable  figures  of  merit  such  as  UTBF,  MTTR,  BIT  effectiveness,  etc.  The  exact 
figure  of  merit  to  be  used  is  dependent  on  the  approach  found  feasible  and  the  degree  to 
which  information  sources  exist  which  support  evaluation.  Four  approaches  were 
investigated  and  one  of  these  was  selected  to  develop  a  preliminary  model. 

Ideally,  the  method  of  developing  Rif  VI  Task  Benefit-Indices  is  to  make  a 
quantitative  assessment  of  the  benefit  prior  to  and  after  application  of  each  individual 
task.  The  R 3c  VI  task  benefit  would  then  be  the  gain  in  benefit  experienced  by  applying 
the  task.  Aside  from  the  availability  of  such  data,  this  method  is  over  simplified,  and 
suffers  from  many  inherent  difficulties.  For  example,  FMEAs  can  be  done  at  various 
system  and  subsystem  levels,  so  that  the  benefit  gained  from  performing  an  FMEA  is 
not  just  a  function  of  whether  or  not  the  F  VIE  A  was  applied,  but  also  the  level  of 
application  (i.e.,  "how  much"  F.VIEA  is  applied).  Also,  while  many  R&M  tasks  are  only 
quantifiable  as  either  being  present  or  not,  the  benefit  gained  by  applying  such  a  task 
could  depend  on  other  factors  as  well,  including  equipment  complexity,  and  the 
presence  of  other  RAM  tasks.  A  reliability  modeling  effort,  for  example,  would  make 
very  little  sense  without  the  associated  prediction  effort  which  provides  the  input  data 
used  by  the  models.  Thus,  it  is  difficult  to  accurately  assess  the  benefit  gained  from 
each  task  separately  either  quantitatively  or  subjectively.  It  is  tacitly  assumed  in  the 
approaches  described  below,  therefore,  that  R&M  tasks  are  to  be  grouped  appropriately. 

4.1.1  "Task-By-Task"  Assessment  Based  on  Hardware  Engineering  Changes.  The 
reliability  and  maintainability  impact  of  the  hardware  changes  caused  by  the  individual 
task,  or  composite  set  of  tasks,  is  assessed  to  determine  an  appropriate  change  in  the 
figure  of  merit  associated  with  the  task(s).  This  method  is  used  for  each  task  or  group 
of  tasks  throughout  the  program  until  a  point  where  the  reliability  and  maintainability 
characteristics  of  the  equipment  have  been  reasonably  established,  usually  at  the 
conclusion  of  qualification  testing. 

The  primary  advantage  of  a  task-by-task  assessment  of  this  type  is  that  the 
approach  is  direct  and  traceable.  The  assessment  of  an  improvement  in  equipment 
reliability  or  maintainability  is  dependent  on  evidence  of  a  design  change  which  was 
caused  by  the  results  of  the  task.  The  assessment  of  changes  caused  by  a  task  requires 
that  detailed  engineering  records  be  available  throughout  the  development  program. 

This  level  of  detail  is  potentially  realizable  for  a  development  program  when  hardware 
changes  are  formally  documented  in  engineering  change  proposals  (ECPs)  and  in  test 
reports. 
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Hio  main  disadvantage  found  with  a  direct  task-by-task  assessment  is  that  the 
amount  of  program  data  and  level  of  detail  required  to  use  the  method  is  prohibitive. 
Engineer  notebooks,  laboratory  record  Dooks,  and  deliverable  data  items  are  normally 
available  from  the  early  equipment  development  period  but  these  records  do  not  contain 
sufficient  detail  to  allow  the  large  number  of  informal  changes  which  occur  as  a  result 
of  RAM  analyses  to  be  identified  or  evaluated.  Additionally,  the  direct  task  analysis 
approach,  which  necessarily  relies  on  engineering  judgment  for  assessment  of  changes 
due  to  RAM  task  performance  (as  opposed  to  other  reasons)  would  be  subjective  and 
unique  to  Hughes.  Therefore,  a  task-by-task  assessment  approach  is  not  recommended 
as  a  basis  for  developing  a  benefits  relationship. 

4.1.2  Task  Assessment  by  Case  Study.  Case  studies  which  investigate  the  benefit 
of  tasks  in  an  RAM  program  tend  to  be  subjective  but  are  useful  as  general  support  for 
more  quantitative  analyses.  The  Institute  for  Defense  Analysis  (IDA)  has  conducted  a 
number  of  these  studies  over  the  past  several  years.  Although  the  RAM  benefits  for  a 
given  case  are  somewhat  unique  to  the  situation,  an  assessment  across  a  number  of 
programs  should  identify  the  major  benefiting  tasks.  The  following  paragraphs 
summarize  six  IDA  case  studies  which  had  applicable  RAM  information: 

F-15  AN/APG  83  R  ADAR.  This  case  study  [3]  also  found  that  strong  customer  and 
program  office  management  support  are  necessary  for  an  effective  RAM  program.  The 
study  questioned  the  value  of  the  "Laboratory"  R  Demo  and  found  that  "failure-free" 
burn-in/acceptance  testing  to  be  effective.  Further,  a  field-FRACAS  program  was 
judged  to  be  the  most  effective  tool  for  identifying  and  correcting  reliability  problems. 

F/A-18,  AN/APG-85  RADAR.  This  study  [4]  did  an  item  by  item  qualitative/ 
quantitative  analysis  of  the  RAM  Tasks  performed  during  this  program.  The  tasks 
which  provided  the  "most  return  for  the  money"  were  TAAF,  FRACAS,  FRB  and  the 
worst  case/stress  analyses.  The  study  was  critical  of  the  100%  piece  part  level  FMECA 
as  not  effective.  However,  the  study  did  find  value  in  the  FMECA  as  a  tool  at  other 
levels. 

F-16  AN/APG-66  RADAR.  This  case  study  [5]  pointed  out  that  strong  management 
support  from  both  the  customer  and  program  office  are  necessary  for  an  effective  RAM 
program.  The  "lessons  learned"  found  that  ESS  at  all  levels  of  test  to  be  very 
effective.  As  in  the  previous  case,  TAAF,  FRACAS  and  FRB  were  again  identified  as 
effective. 

T700  Jet  Turbine  Engine.  The  primary  factor  this  study  [6]  pointed  out  was  that 
reliability  and  maintainability  must  be  "designed-in".  The  lessons  learned  dealt  with 
quality  factors  more  than  purely  R  or  M  factors.  However,  vendor  control  was  deemed 
to  be  very  necessary. 

AN/APN-128  Lightweight  Doppler  Navigation  System.  This  case  study  [7]  also 
stressed  the  need  for  management  involvement  for  an  RAiM  program  to  be  successful. 
The  RAM  tasks  found  to  be  effective  were  the  design  analyses  such  as  worst  case,  and 
thermal  stress  (hot  and  cold)  analyses.  ESS  testing  at  all  levels  and  FRACAS  was  also 
considered  valuable. 
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FIKEFINDEK  Weapons  Locating  Radar.  The  FIREFINDER  case  study  [8]  found  the 
FRB,  TAAF  and  R  Growth  tasks  to  be  effective.  However,  FRACAS  was  seen  as 
ineffective  in  the  production  phase  of  the  program  because  of  the  lack  of  ability  to 
effect  change  in  the  factory  and  because  it  tended  to  get  mired  in  minutia  which 
masked  the  "big  swingers". 


4.1.3  Task  Assessment  by  Expert  Opinion.  The  relative  benefit  obtained  by  the 
performance  of  various  R&M  program  tasks  can  also  be  assessed  through  the  use  of 
expert  opinion  (e.g.,  as  obtained  from  an  industry  survey.)  The  main  advantage  to  this 
approach  is  that  the  assessment  is  simple  to  perform  and  provides  an  industry-wide 
input  from  the  R3cM  community.  Additionally,  the  expert  opinion  method  may  provide 
an  industry  concensus  with  the  results  of  other  approaches. 


The  disadvantages  to  this  approach  are  that  it  is  an  entirely  subjective,  and  usually 
non-qualitative  approach,  and  there  is  also  a  risk  of  weak  response  to  the  survey.  The 
expert  opinion  method  of  assigning  R&M  benefit  changes  to  a  specific  task  where 
quantitative  data  is  weak,  however,  could  be  useful  as  a  rough  check  on  the 
reasonableness  of  the  results  of  other,  more  objective  methods. 


4.1.4  Task  Assessment  by  Linear  Regression.  The  approach  which  showed  the  most 
promise  in  producing  a  quantitative  result  is  through  the  use  of  linear  regression 
analysis  techniques.  This  approach  has  the  advantage  of  being  the  most  objective, 
requiring  only  an  estimate  of  the  total  benefit  (e.g.,  the  observed  MTBF)  from  each 
program  and  is  consistent  with  the  CER  development.  Additionally,  the  results  of  this 
method  can  be  compared  against  the  results  of  case  studies  and  an  industry  expert 
opinion  survey  for  consistency. 


The  only  disadvantage  associated  with  this  approach  is  that  the  sample  size  must 
be  sufficiently  large  to  provide  reasonable  estimating  accuracy. 


‘‘uVn 


5.0  CONCLUSIONS  AND  RECOMMENDATIONS 

A  method  of  estimating  R&M  task  cost  (i.e.,  labor  hours)  based  on  data  that  is 
normally  available  during  the  early  program  planning  stages  has  been  developed.  The 
method  consists  of  a  set  of  six  cost-estimating-  relationships  (CER's)  which  were 
derived  from  data  on  23  projects  using  multiple  linear  regression  (MLR)  techniques. 

The  CER's  provide  a  tool  for  comparing  alternate  R&M  program  costs  and  guidelines 
for  estimating  what  a  specific  RJcM  program  should  cost.  Many  of  the  M1L-STD  785 
and  470  tasks  required  grouping  in  order  to  provide  sufficient  data  to  generate  these 
CER's.  A  summarization  of  these  groupings  is  provided  in  Table  1.2-1. 

Several  models  are  provided  for  each  CER  and  those  models  consisting  of  the 
largest  number  of  variables  have  the  highest  values  and  generally,  provide  more 
accurate  estimates.  Tabulations  of  each  CER  are  also  provided  using  the  model  with 
the  highest  R^  value  and  should  be  applicable  to  most  situations.  The  ranges  of 
application  of  these  models  and  tables  (i.e.,  the  ranges  of  values  of  the  independent 
variables)  and  the  identification  of  associated  labor  not  accounted  for  in  the  estimates 
should  be  carefully  noted  in  the  detailed  description  of  each  CER  (Section  3.0).  The 
data  base  used  to  generate  the  CER's  can  be  expanded  by  adding  data  points  to  the 
input  data  given  in  Appendix  B  (Tables  B.6-1  through  B.6-6)  and  adhering  to  the 
constraints  on  the  variables  given  under  the  appropriate  subsections  of  3.0.  The 
structure  of  the  models  developed  should  be  sufficiently  robust  to  be  applicable  in  an 
expanded  data  base. 

The  benefits  derived  from  implementing  the  various  tasks  in  an  R<5cM  program  are 
very  difficult  to  measure  directly  except  in  special  cases  where  the  performance  of  a 
specific  task  (e.g.,  FMECA)  led  to  the  generation  of  identifiable  cost-saving  design 
changes.  These  before-and-after  assessments  are  unique  to  each  program  and  normally 
cannot  be  generalized. 

For  developing  a  usable  benefits  model  in  a  follow-on  study  (i.e.,  one  with  a 
significantly  wider  application)  a  much  larger  sample  size  would  be  required.  A  sample 
size  of  30  or  more  systems  is  recommended.  In  addition,  industry  questionnaires  and 
case  studies  (both  existing  and  new)  should  be  included  as  supporting  data  for  the  model 
development  in  a  follow-on  study.  Similarly,  the  long  term  (or  life  cycle)  benefits  of  an 
R&M  program  such  as  the  benefits  derived  in  reduced  spares,  maintenance  manpower 
and  increased  operational  readiness  should  not  be  ignored.  For  example,  a  small  gain  in 
actual  MTBF  per  system  can  have  a  substantial  life  cycle  cost  impact  in  these  areas  on 
a  large  deployment  of  such  systems. 


F1REF1NDER  Weapons  Locating  Radar.  The  F1REF1NDER  case  study  [8]  found  the 
FRB,  TAAF  and  R  Growth  tasks  to  be  effective.  However,  FRACAS  was  seen  as 
ineffective  in  the  production  phase  of  the  program  because  of  the  lack  of  ability  to 
effect  change  in  the  factory  and  because  it  tended  to  get  mired  in  minutia  which 
masked  the  "big  swingers". 

4.1.3  Task  Assessment  by  Expert  Opinion.  The  relative  benefit  obtained  by  the 
performance  of  various  R&M  program  tasks  can  also  be  assessed  through  the  use  of 
expert  opinion  (e.g.,  as  obtained  from  an  industry  survey.)  The  main  advantage  to  this 
approach  is  that  the  assessment  is  simple  to  perform  and  provides  an  industry-wide 
input  from  the  R&M  community.  Additionally,  the  expert  opinion  method  may  provide 
an  industry  concensus  with  the  results  of  other  approaches. 

The  disadvantages  to  this  approach  are  that  it  is  an  entirely  subjective,  and  usually 
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expert  opinion  method  of  assigning  R3c  iVl  benefit  changes  to  a  specific  task  where 
quantitative  data  is  weak,  however,  could  be  useful  as  a  rough  check  on  the 
reasonableness  of  the  results  of  other,  more  objective  methods. 

4.1.4  Task  Assessment  by  Linear  Regression.  The  approach  which  showed  the  most 
promise  in  producing  a  quantitative  result  is  through  the  use  of  linear  regression 
analysis  techniques.  This  approach  has  the  advantage  of  being  the  most  objective, 
requiring  only  an  estimate  of  the  total  benefit  (e.g.,  the  observed  MTBF)  from  each 
program  and  is  consistent  with  the  CER  development.  Additionally,  the  results  of  this 
method  can  be  compared  against  the  results  of  case  studies  and  an  industry  expert 
opinion  survey  for  consistency. 

The  only  disadvantage  associated  with  this  approach  is  that  the  sample  size  must 
be  sufficiently  large  to  provide  reasonable  estimating  accuracy. 
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method  consists  of  a  set  of  six  cost-estimating-  relationships  (CER's)  which  were 
derived  from  data  on  23  projects  using  multiple  linear  regression  (MLR)  techniques. 
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the  highest  R^  value  and  should  be  applicable  to  most  situations.  The  ranges  of 
application  of  these  models  and  tables  (i.e.,  the  ranges  of  values  of  the  independent 
variables)  and  the  identification  of  associated  labor  not  accounted  for  in  the  estimates 
should  be  carefully  noted  in  the  detailed  description  of  each  CER  (Section  3.0).  The 
data  base  used  to  generate  the  CER's  can  be  expanded  by  adding  data  points  to  the 
input  data  given  in  Appendix  B  (Tables  B.6-1  through  B.6-6)  and  adhering  to  the 
constraints  on  the  variables  given  under  the  appropriate  subsections  of  3.0.  The 
structure  of  the  models  developed  should  be  sufficiently  robust  to  be  applicable  in  an 
expanded  data  base. 

The  benefits  derived  from  implementing  the  various  tasks  in  an  R<5cM  program  are 
very  difficult  to  measure  directly  except  in  special  cases  where  the  performance  of  a 
specific  task  (e.g.,  FMECA)  led  to  the  generation  of  identifiable  cost-saving  design 
changes.  These  before-and-after  assessments  are  unique  to  each  program  and  normally 
cannot  be  generalized. 

For  developing  a  usable  benefits  model  in  a  follow-on  study  (i.e.,  one  with  a 
significantly  wider  application)  a  much  larger  sample  size  would  be  required.  A  sample 
size  of  30  or  more  systems  is  recommended.  In  addition,  industry  questionnaires  and 
case  studies  (both  existing  and  new)  should  be  included  as  supporting  data  for  the  model 
development  in  a  follow-on  study.  Similarly,  the  long  term  (or  life  cycle)  benefits  of  an 
R<5cM  program  such  as  the  benefits  derived  in  reduced  spares,  maintenance  manpower 
and  increased  operational  readiness  should  not  be  ignored.  For  example,  a  small  gain  in 
actual  MTBF  per  system  can  have  a  substantial  life  cycle  cost  impact  in  these  areas  on 
a  large  deployment  of  such  systems. 
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APPENDIX  A  -  APPROACH  TO  MODEL  FITTING 


Multiple  linear  regression  (MLR)  was  used  to  analyze  the  data.  This  was  deemed 
the  most  effective  method  for  determining  which  aspects  of  a  reliability  program  have 
significant  effects  on  costs  and  for  producing  accurate  cost  estimating  relationships 
(CER's).  The  basis  assumptions  of  this  technique  are: 

1.  The  dependent  variable,  Y  (labor  hours),  is  an  intrinsically  linear  combination 
of  p  independent  variables  (hardware  complexity,  number  of  items,  reporting 
formality).  This  relation  is  called  the  regression  equation,  and  matrix 
notation  the  model  is  represented  in  the  form 


Y  =  X  B  +  e 


where 


A 


Y  is  an  (n  x  1)  vector  of  the  n  observations  of  the  dependent  variable 
(labor  hours) 

X  is  an  (n  x  p)  matrix  of  observations  of  the  independent,  variables 

e  is  an  (n  x  1)  vector  of  errors. 

2.  The  elements  ej  1  <_  i  <_  n  of  the  vector  e  represent  values  of  a  normally 
distributed  random  variable.  This  assumption  is  reasonable  since  the  error 
term  is  most  probably  the  sum  of  errors  from  a  large  number  of  sources  and, 
therefore,  by  the  Central  Limit  Theorem,  their  sum  will  have  a  distribution 
that  will  be  approximately  normal,  regardless  of  the  type  of  probability 
distribution  the  separate  error  components  may  have. 

3.  The  expected  value  of  e  is  E  (e)  =  (o)  and  the  variance  is  V  (e)  =  I  a2,  where  I 
is  the  identity  matrix,  so  the  elements  of  e  are  uncorrelated.  That  is,  E  (Y)  = 
XB. 

The  above  imply  that  the  error  sum  of  squares  is: 

e'e  =  (Y  -  XB)'( Y  -  XB)  =  Y'Y  -B’X'Y  -Y'XB  +  B'X'XB 
=  Y'Y  -  2Y'XB  +  B'XB 

The  least  squares  estimate  of  B  produces  the  least  possible  value  of  e'e.  By 
differentiating  with  respect  to  B,  setting  the  resulting  equation  to  zero  and  replacing  B 
by  b,  the  so-called  normal  equation  results 

(X'X)b  =  X'Y 

b  =  (X'X)-lX’Y 

This  solution,  b,  called  the  least  squares  estimate  of  B,  is  the  best  linear,  unbiased 
estimate.  Further  details  are  given  in  Draper  and  Smith. 
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Computer  programs  described  below  used  this  method  to  obtain  estimates  of  B. 


A. I  Intrinsically  Linear  Models.  Models  which  are  not  linear  but  are  "intrinsically 
linear"  may  be  made  linear  (examples  are  illustrated  in  Figure  A.  1-1)  by  appropriate 
changes  of  variables.  Two  of  the  most  common  of  these  are  the  exponential  model  and 
the  power  function  model. 


An  exponential  model 


Y  =  b0e 


blXl  + 


b2x 


2 


can  be  made  linear  by  taking  natural  logarithms  of  both  sides  to  obtain: 
In  Y  =  In  bQ  +  bj  Xj  +  b£  X2 

a  power  function  model 


Y  =  b0Xl 


2 


can  also  be  converted  to  a  linear  form  by  taking  natural  logarithms  of  both  sides: 
In  Y  =  In  bg  +  bj  In  X^  +  b2  In  X2 


Table  A. 1-1  contains  the  matrix  forms  after  transformation  of  these  nonlinear 
models  to  linear  models. 


TABLE  A.  1-1.  EXAMPLE  NON-LINEAR  MODELS 


These  two  nonlinear  model  are  typical  of  the  types  of  models  that  were  fitted  to 
the  data. 


A. 2  Computer  Methods.  Multilinear  regression  analysis  was  performed  using  the 
Statistical  Analysis  System  (SAS)  multilinear  regression  computer  procedures  Cl,  U  and 
STEPWISE  (see  Appendix  B).  The  initial  search  for  acceptable  CEK's  was  performed 
with  STEPWISE.  As  the  name  implies,  the  procedure  examines  a  sequence  of  multiple 
linear  equations  in  a  STEPWISE  manner.  Potential  predictors  (i.e.,  independent 
variables)  are  added  (forward  regression)  or  deleted  (backward  regression)  at  each  step. 
Thus,  a  sequence  of  regression  functions 


Y  =  VVh 


Y  =  b0  +  b!  X1  *  b2  X2 


Y  =  B"  +  b!‘xit" 


+  b "  X 
P  P 


is  produced. 


The  predictors  are  not  added  arbitrarily.  The  program  steers  additions/deletions 
by  statistical  tests.  The  methods  available  within  STEPWISE  include  MAXR, 
FORWARD  AND  BACKWARD.  The  STEPWISE  M  AXR  procedure  was  used  to  develop 
the  basic  CERs  for  all  tasks.  The  other  procedures  were  used  for  checking  and 
verifying  the  models  obtained  by  use  of  MAXR.  The  SAS  procedure  GEM  was  used  to 
obtain  additional  statistical  information  about  the  CERs  that  were  otherwise 
acceptable. 


MAXR  or  Maximum  R2  improvement  is  considered  to  be  superior  to  FORWARD 
and  BACKWARD  described  below  and  almost  as  good  as  all  possible  regressions.  This 
method  does  not  fix  on  a  single  model.  Rather,  it  searches  for  the  best  one-variable 
model,  the  best  two-variable  model,  etc.  Initially,  the  one-variable  model  producing 
the  highest  R2  is  found.  Of  the  remaining  variables,  the  one  that  would  give  rise  to  the 
greatest  increase  in  R 2  is  included.  The  important  aspect  of  this  method  is  that  it  is 
not  assumed  to  be  the  best  two-variable  model.  Each  variable  in  this  two-variable 
model  is  compared  to  each  variable  not  included  in  the  model.  All  possible  switches  in 
variables  are  compared  and  the  one  producing  the  greatest  increase  in  R2  is  made.  The 
process  continues  until  no  switch  could  increase  R2.  The  two-variable  model  thus 
obtained  is  reported  as  the  best  two-variable  model  the  technique  can  determine.  The 
comparing-and-switching  process  is  repeated  for  each  additional  variable.  MAXR  thus 
has  the  property  that  all  variable  switches  are  evaluated  before  any  changes  are  made. 
This  is  an  improvement  over  procedures  in  which  the  worst  variable  may  be  removed 
without  consideration  of  the  effect  of  adding  the  best  remaining  variable. 


In  FORWARD,  the  first  step  is,  again,  to  find  the  one-variable  model  that  shows 
the  greatest  R2  value.  For  each  of  the  remaining  independent  variables,  F-statistios 
are  computed.  These  reflect  the  variable's  contribution  to  the  model  if  it  wore 
included.  The  variable  which  has  the  largest  F-value  is  added  to  the  model  (provided  it" 
significance  level  is  above  a  certain  predetermined  threshold.)  The  process  continues 
and  variables  are  added  one  at  a  time  until  no  remaining  variable  yields  a  significant 
F-statistic.  Once  a  variable  is  added  to  the  model,  however,  it  is  not  removed. 


a 


i: 
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In  BACKWARD,  statistics  are  calculated  for  a  model  including  all  the  independent 
variables.  These  predictors  are  removed  one  by  one  until  all  remaining  variables 
produce  F-statistics  above  a  given  significance  level. 

A. 3  Choice  of  Useful  Models.  The  results  in  the  last  section  described  the 
computerized  tools  that  were  used  to  evaluate  the  various  models.  Generally,  more 
than  one  potentially  useful  model  was  obtained  for  each  R/M  task.  These  models  differ 
in  the  number  of  type  of  inputs  required  for  their  use.  It  is  recommended  that  CERs 
exhibiting  the  best  fit  (i.e.,  highest  R^  value)  be  used  whenever  possible.  But  if  it  is  not 
possible  to  get  all  the  inputs  for  the  best  predictor,  the  remaining  models  should  be 
selected  in  order  of  highest  R^  value. 

In  order  to  be  considered  effective,  CER  a  model  was  required  to  explain  at  least 
90%  of  the  total  variation  of  the  regression  (i.e.,  R^  =  .90)  and  be  statistically 
significant  at  the  95%  level. 

A. 4  Plausibility  Checks.  It  is  important  to  note  that  the  final  model  selections 
(or  CERs)  do  more  than  provide  statistically  significant  predictors.  At  all  stages  of  the 
process,  candidates  models  were  examined  by  reliability  experts  to  ensure  plausibility 
and  usability.  Checks  were  made  to  ensure  that  important  program  inputs  known  to 
relate  to  task  cost  were  included.  In  addition,  various  sets  of  reasonable  values  of  the 
independent  variables  were  input  using  each  CER  to  determine  if  the  outputs  were  also 
reasonable.  These  efforts  helped  to  ensure  the  usefulness  of  predictions  made  using  the 
CERs. 


* 
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APPENDIX  B  -  REGRESSION  ANALYSIS  RESULTS 


The  following  paragraphs  define  the  statistics  used  in  the  analyses,  the  input  data 
from  the  projects  used,  and  provide  the  computer  printouts  of  the  results  that  were 
used  to  develop  the  R&M  task  cost-estimating-relationships  (CF.Rs) 

B.l  R  -  Statistic.  R  is  callev  t he  multiple  correlation  coefficient. 

The  objective  of  the  linear  least  squares  procedure  is  to  account  for  as  much  of 
the  total  variability  of  the  data  as  possible  by  means  of  the  fitted  equation. 

Consider  the  following  identity: 

(B.  1-1)  Y.  -  Y.=  (Y  •-  Y)  -  Y- Y) 

11  11 

where 

Y.is  observed  labor  hours  at  ith  observation 

Y.  indicates  labor  hours  as  estimated  bv  the  CER  at  ith  set  of  conditions. 
i 

Y  is  the  mean  of  the  Y. 

Thus  the  residual  e.=  Y.  -  Y.  is  the  difference  between: 

a.  The  deviation  of  the  observed  Y.  from  the  overall  mean,  Y,  and 

l  ’ 

A  _ 

b.  The  deviation  of  the  fitted  Y.  from  Y,  the  mean  of  the  observed  values. 

l  ’ 

He  can  rewrite  (B.l-1)  as 

(Y  .-  Y)  =  (Y  -  Y)  +  (Y  .-  Y). 

If  we  square  both  sides  and  sum  i  =  1,  2,  ...  n,  we  obtain 
a  9  n  ^  o  a 

(B.l- 2)  l  (Y  -  Y)  =  l  (Y  -  Y)  +  l  (Y  -  Y  ) 
i=l  i  i=l  i=l  i  i 

(It  is  shown  in  Draper  and  Smith  [2]  that  the  cross-product  term  vanishes.) 

The  quantity  (Y  -  Y)  is  the  deviation  of  the  ith  observation  from  the  overall 
mean.  So  the  left  hand  side  of  (B.  1-2)  is  the  sum  of  squares  of  deviations  of  the  obser¬ 
vations  from  the  mean;  shortened  to  SS  about  the  mean.  It  is  also  called  the  corrected 
sum  of  squares  of  the  Y's.  As  above  Yi  -  Y  is  the  deviation  of  the  predicted,  or  fitted, 
value  of  the  ith  observation  from  the  mean,  and  Yj  -  Y  is  the  deviation  of  its 
observation  from  its  predicted  value  (i.e.,  the  ith  residual),  we  can  express  (B.  1-2)  in 
words): 


I  Sum  of  Square  \  _  /Sum  of  Square  \  +  /Sum  of  Squares 

\  About  the  Mean  )  (Due  to  Regression)  l  about  Regression 

This  indicates  that  a  way  of  determining  the  usefulness  of  the  regression  line  as  a 
predictor  is  to  compare  how  much  of  the  SS  about  the  mean  is  in  the  SS  due  to 
regression  and  how  much  is  in  the  SS  about  regression.  We  desire  that  the  SS  due  to 
regression  be  much  greater  than  the  SS  about  regression,  or  that 

p  2  _  SS  due  to  regression 
SS  about  mean 

be  as  close  as  possible  to  one. 

B.2  Analysis  of  Variance  Table.  The  above  sums  of  squares,  like  any  sum  of 
squares,  have  associated  degrees  of  freedom.  This  represents  the  number  of 
independent  pieces  of  information  involving  the  n  independent  numbers  Yj,  Y2  •••> 

Yn  needed  to  compute  the  sum  of  squares.  Consider  Y.  -  Y,  Y„-  Y,  ...,  Y  -  Y. 

Only  (n-1)  are  independent  since  all  n  of  them  sum  to  zero  by  definition  oV  the  mean. 
Thus,  SS  about  the  mean  has  (n-1)  degrees  of  freedom. 

The  number  of  degrees  of  freedom  for  SS  due  to  regression  is  equal  to  the  number 
of  coefficients  determined  by  the  fitting  (not  including  the  intercept).  Each  bj,  i_>  1 
is  a  function  of  Y Y 2,  ...,  Yn.  Also 

I  (Yj  -  Y)  2  =  l  [b0  +  bi  Xi  +  ...  +  bp  Xp)  -  (b0  +  bj  X  ...  bp  X] 2 

So  there  are  only  p  independent  pieces  of  information  involving  the  Yj  needed  to 
compute  SS  due  to  regression. 

Now,  by  subtraction,  the  SS  about  regression  has  (n-1)  -  p  degrees  of  freedom 
(df).  The  SS  about  regression  is  also  called  the  residual  sum  of 

squares  (since  Y.  -  Yj  are  the  residuals). 

The  computed  error  mean  square  is  the  statistic  s^,  used  as  an  estimate  of  the 
assumed  fixed  but  unknown  6  2 ,  variance  of  the  error  term  of  variance  about 
regression.  The  variance  about  regression  gives  a  measure  of  the  error  involved  in 
predicting  an  observed  value  of  Y  from  a  given  value  of  X  using  the  determined  model. 

B.3  F-Test  for  Significance.  The  F-test  was  used  to  eliminate  CER's  that  were 
not  statistically  significant.  Consider  the  general  CER  defined  by 

y  =  b0  +  bj  Xj  ...  +  bp  Xp  +  e. 

This  CER  was  eliminated  from  further  study  if  the  hypothesis 
H0  :  bj  =  0  i  =  l,  2,  ...,  p 

(i.e.,  hypothesis  that  all  coefficients  should  be  zero)  could  not  be  rejected  at  the  95% 
level. 


The  test  statistic  is 


U  Y  Y)  2/p 
X  ( Y.  -Y.)2/(n-p-l) 


Mean  Square  Regression 
Mean  Square  Error 


If  F  >  Fa(p,n-p-l)  for  an  F  statistic  with  a  =  0.05  and  the  given 

degrees  of  freedom,  H0  was  rejected  and  the  CER  was  considered  significant. 

On  the  computer  printout  (see  B.6. 1),  Prob  >  F  gives  1  —  ot  so  if  it  was  less  than  0.05  the 
CE  R  was  taken  as  significant.  If  it  was  greater  than  0.05  the  CE  R  was  eliminated. 


B.4  Analysis  of  Residuals.  Recall  that  the  residuals  are  defined  as  the  n 

difference  e.=  Y  . -  Y.,  i  =  1,  2,  ...,  n.  As  before,  Y.  is  an  observation  and  Y.  is  the 

corresponding  predicted  value  obtained  by  use  of  the  fitted  regression  equation.  Thus, 
the  residuals,  ej,  are  the  difference  between  what  is  observed  and  what  the  model 
predicted.  We  can  also  think  of  the  ej  as  the  observed  random  errors  if  the  model  is 
correct.  In  performing  regression  analysis  we  have  made  the  usual  assumptions  that  the 
errors  are  independent,  have  zero  mean,  a  constant  (although  unknown)  variance,  and 
follow  a  normal  distribution.  Tests  were  made  to  determine  if  the  residuals  conformed, 
or  at  least  did  not  deny,  these  assumptions. 


Draper  and  Smith  [2]  discuss  plotting  e^  vs  Y  to  see  if  a  normal  distribution  with 
mean  zero  and  constant  variance  is  reasonable.  Figures  B.4-1  through  B.4-7  provide 

a 

ejvs.  Y  plots  for  each  CER  task  and  for  the  benefits  relationship.  Although  some  of 
the  plots  indicate  a  downward  slope,  the  hypothesis  of  randomness  could  not  be 
rejected.  This  conclusion  is  further  supported  by  analyzing  the  sign  runs  (see  B.5). 


B.5  Analyzing  Sign  Runs  in  the  Residuals.  The  pattern  of  signs  (positive  or 
negative)  of  the  residuals  was  examined  for  each  fit.  For  example,  the  pattern  of  signs 
of  the  residuals  for  R<5rM  Program  Plan  (Tj q  j )  is 

(+)  (-  -)  (+)  (-)  (+  +  +  +)  (-  -)  (+). 

There  are  nj  =  1  plus  signs  and  t\2  =  5  minus  signs.  There  are  u  =  7  "runs"  as  indicated 
by  the  parentheses.  The  probability  of  the  occurrence  of  this  arrangement  if  the 
residuals  truly  have  mean  zero  is  determined  from  [2]  pages  160-lfil.  The  method  for 
nj,  n2  >  10  is  given  on  page  159.  The  probability  of  7  or  fewer  runs  in  the  above 
arrangement  is  .413.  Thus,  on  the  basis  of  this  test,  we  have  no  reason  to  reject  the 
hypothesis  of  randomness  with  mean  zero.  The  "runs"  probability  for  each  CER  is  given 
in  Table  B.5-1  and  indicates  that  the  assumption  of  randomness  has  not  been  violated. 

B.6  Study  Data  Base.  Tables  B.6-1  through  B.6-6  provide  the  data  points  used  in 
generating  the  CER  models.  The  corresponding  computer  outputs  from  the  GLM  and 
STEPWISE  programs  are  given  in  B.7  and  B.8 
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FIGURE  U.4-4.  TASK  203  PREDICTIONS 


fABLE  B.5-1.  ANALYSIS  OF  SIGN  HUNS  IN  THE  KESIHUALS 


CER 

+  Signs 
(n ! ) 

-  Signs 
(n2) 

Runs 

(u) 

PROB  (<_  u) 

T 1 0 1 

6 

5 

4 

0.110 

T 104/5 

4 

8 

7 

0.788 

T  20 1 / 2 

4 

4 

5 

0.6  29 

T  203 

9 

7 

n 

t 

0.231 

r204 

4 

2 

4 

0.800 

T301/3/4 

3 

4 

6 

0.971 

T  R  /  AJ 

10 

12 

11 

0.4  25 

If  PROB  (<  u)  is  less  than  0.05,  randomness  is  rejected 
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TABLE  B.6-5.  INPUT  DATA  FOR  T2(J4 


Project 

No. 

Actual 

Labor 

Hours 

HC 

NOU 

LOD 

204 

Actual 

178 

3 

3 

3 

5  22 

1408 

1 

47 

2 

518 

471 

2 

17 

2 

3  27 

2782 

2 

47 

3 

4  20 

315 

2 

12 

1 

415 

3564 

2 

206 

] 

2 

TABLE  B.6-6.  INPUT  DATA  FOR  T301/3/4 


Project 

No. 

Actual 

Labor 

;  Hours 

NOS 

RQT 

PRAT 

204 

900 

28 

1 

1 

3 

214 

300 

2 

1 

0 

2 

112 

269 

2 

0 

1 

1 

1 24 

318 

20 

1 

0 

1 

1 25 

654 

6 

1 

0 

2 

140 

207 

20 

0 

1 

1 

_  1 

■  585 

3 

-  -  _ 

1 

1 

1 

V 

V 

V 


£ 


'■  i  1 


v 

.V. 


v.wa 


•viv-’^S 


H. 7  Computer  Output  for  (1  LM. 

B.7. 1  Explanation  of  Terms.*  The  OEM  procedure  [12]  produces  the  following 
printed  output  by  default: 

a.  The  overall  analvsis-of-varianee  table  breaks  down  the  CORRECTED 
TOTAL  sum  of  squares  (1)  for  the  dependent  variable  into  the  portion 
attributed  to  the  MODEL  (2)  and  the  portion  attributed  to  ERROR  (3)  . 

b.  The  MEAN  SQUARE  term  is  the  SUM  OF  SQUARES  divided  by  the 
DECREES  OF  FREEDOM  (DF)  (4)  . 

c.  The  MEAN  SQUARE  for  ERROR,  (MS(ERROR)),  is  an  estimate  of  s2,  the 
variance  of  the  true  errors  (7)  . 

d.  The  F  VALUE  (8)  is  the  ratio  produced  bv  dividing  MS(MODEL)  bv 
MS(ERROR). 


A  small  significance  probability,  PR>F,  indicates  that  some  linear 
function  of  the  parameters  is  significantly  different  from  zero.  (9) 


R-SQUARE,  R2,  measures  how  much  variation  in  the  dependent  variable 
can  be  accounted  for  by  the  model  (10). 


C.V.,  the  coefficient  of  variation  (11),  which  describes  the  amount  of 
variation  in  the  population  is  100  times  the  standard  deviation  estimate 
of  the  dependent  variable,  ROOT  MSE,  divided  by  the  MEAN.  The 
coefficient  of  variation  is  often  a  preferred  measure  because  it  is 
unitless. 


h. 


ROOT  MSE  ( 1 2)  estimates  the  standard  deviation  of  the  dependent 
variable  (or  equivalently,  the  error  term)  and  equals  the  square  root  of 
MS(  ERROR). 


i.  M LAN  ( I  3)  is  the  sample  mean  of  the  dependent  variable. 


The  TYPE  1  SS  (14)  measures  incremental  sums  of  squares  for  the  model 
as  each  variable  is  added. 


k. 


The  TYPE  III  SS  (15)  is  the  sum  of  squares  that  results  when  that  variable 
is  added  last  to  the  model. 


This  section  of  the  output  gives  the  ESTIMATES  (18)  for  the  model 
P  AR  AME TERs-the  intercept  and  the  coefficients. 


m.  1  FOR  IK):  PARAMETER  =  0  ( 1 7)  is  the  Student 's  t  value  for  testing  the 
null  hypothesis  that  the  parameter  (if  it  is  estimable)  equals  zero. 


*  I  he  numbers  m  parentheses  refer  to  the  sample  output. 
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n.  The  significance  level,  PR>iTi  ,  (18)  is  the  probability  of  getting  a  larger 
value  of  t  if  the  parameter  is  truly  equal  to  zero.  A  very  small  value  for 
this  probability  leads  to  the  conclusion  that  the  independent  variable 
contributes  significantly  to  the  model. 

o.  The  STD  ERROR  OF  ESTIMATE  (19)  is  the  standard  error  of  the 
estimate  of  the  true  value  of  the  parameter. 

p.  Observed  values  (20)  of  the  dependent  variable,  Y. 

q.  Predicted  values  (21)  of  the  dependent  variable,  Y.  Uses 
estimates  from  (16)  in  model. 

r.  Residual  (22)  or  e.,  Y  -  Y. 

s.  90%  confidence  intervals  (23)  (24)  for  Y. 


TABLE  B.7.1-1.  SAMPLE  OUTPUT 

TASK  101  RIM  PROGRA. ■  PLAN  ll:*5  TLCSOAY >  DECEMBER  J.  1SB5  2 

GENERAL  LINEAR  MODELS  PROCEDURE 


OtPEWjENT  VARIABLE.  f»«R 


SOURCE 

or 

SUM  or  SQUARES  (1) 

EC  AN  SQUARE 

F  VALUE 

PR  >  F 

fc -SQUARE 

C.  V. 

MOD  Ft 

3 

E8S.177ER1B4  12) 

96.05*0*728 

82.50  (8) 

0.0001  (9) 

0.468687(10) 

20  -  9446  (11) 

ERROR 

a 

S.n5510<*0  (3) 

1.16441386  (7) 

ROOT  USE 

WHR  MEAN 

uncorrected  total 

n 

JR7.<>RZ«OZ7<. 

1.07*08010  (12) 

5- 15084834  (13) 

SOURCE 

DF 

TYPE  I  SSI  1  4) 

F  VALUE  PR  >  F 

OF 

TYPE  III  SS( 

15)  r  value 

PR  >  F 

DC 

1 

1P1  87S01S4R 

145.6*  0.0001 

1 

0.51581131 

0.44 

0.5252 

NT 

1 

1  IS  .  BPS 581.57 

99.53  0.0001 

1 

46.65666261 

40.07 

0.0002 

«r 

1 

7.<.058P185 

2.06  0.1887 

1 

2.40389163 

2.06 

0. 18*7 

T  TOR  HO.  PR  >  ITI 

STO  ERROR 

OF 

parafcter 

ESTIMATE  (  16) 

PARAfC TER  =  0  (  1  7) 

(18)  ESTIMATE  (19) 

oc 

0  53200134 

0.66 

0  5252 

0.80067471 

1  704*8666 

6.35 

0  0002 

0.27004334 

RF 

1  33563076 

1.44 

0.1807 

0.92957114 

OBSERVATION 

OOSE  RVED 

PREOICTEO 

RESIDUAL 

LCBCR  90/  CL 

UPPER  90/  CL 

VALUE 

VALUE 

INDIVIDUAL 

IFCIVIOUAL 

(20) 

<?1) 

(22) 

123) 

1 

3.55554806 

4  0*249250 

-1.30714424 

2. *4594888 

7.07*03572 

z 

5  12  346  348 

4  *2144441 

0  50251*55 

2 . 38884348 

6  65404538 

l 

5  3844*506 

4  941*3580 

0  44285*2* 

2.4*493*36 

7.41833223 

4 

5  480*3642 

5.42817592 

0  0522*300 

3  05301*51 

7.60373234 

£ 

4  1108758* 

2  3  701  31  74 

1.  74074212 

0.24613*07 

4.4*412741 

* 

5  3 10 1 1 444 

6.03400 74] 

-0  71**6741 

38*155266 

8  20806216 

7 

4  476733  74 

5. *1853713 

-0  6418033* 

3  23457556 

8  0024*8  70 

0 

*  077*4224 

*  13845**5 

-0 . 0*06143* 

3  *664*184 

8  31042143 

4 

5  484  7**45 

*  210508*1 

-0  725711*7 

3. 8**6**8* 

6  521317*2 

10 

6  05912320 

4  84974154 

1  2043616* 

2.631876*0 

7  06760617 

11 

5  00  754*54 

5  90915*55 

1  0*645*61 

1  7*733  761 

*  180*41*5 

I 

a 


1 


1 


$ 

V/v 

P 


Km< 


'iun  or  resiouais 

SUM  or  SQU  A  RIO  Rf  SI  DUAL  S 

sum  or  squared  resiouais  -  irro* 

PRESS  STATISTIC 

FIRST  ORDER  AUTOCORRELATION 

DURBIN  HA T SON  0 

B.7.2  Output  of  rSLM. 
printouts  for  f*LM. 


I  5*4060** 
*  315310*0 
•0  00000000 
IS  40444475 
0  0*1*0167 

l  bio?s7ro 


Tables  B.7.2 -1  through  R.7.2-7  provide  the  computer 
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B.8  Computer  output  for  STEPWISE 


B.8.1  Explanation  of  Terms.*  For  each  model  of  a  Riven  size,  STEPWISE  [12J 
Prints  an  analvsis-of-variance  table,  the  regression  coefficients,  and  related  statistics. 
Hie  analvsis-of-variance  table  includes: 

a.  The  source  of  variation  REGRESSION  (1)  ,  which  is  the  variation  that  is 
attributed  to  the  independent  variables  in  the  model  the  source  of 
variation  ERROR  (2)  ,  which  is  the  residual  variation  that  is  not 
accounted  for  by  the  model,  and  the  source  of  variation  TOTAL  (3)  , 
which  is  corrected  for  the  mean  of  v  if  an  intercept  is  included  in  the 
model,  uncorrected  if  an  intercept  is  not  included. 

b.  DF,  degrees  of  freedom  (4) 

c.  SUMS  OF  SQUARES  for  REGRESSION,  ERROR,  and  TOTAL  (5) 

d.  MEAN  SQUARES  for  REGRESSION  and  ERROR  (6) 

e.  The  F  value  (7)  which  is  the  ratio  of  the  REGRESSION  mean  square  to 
the  ERROR  mean  square. 

f.  PROB  >  F  (8)  ,  the  significance  probability  of  the  F  value 

g.  R  SQUARE  (9)  or  R 2,  the  square  of  the  multiple  correlation  coefficient. 

h.  C(P)  statistic  proposed  by  Mallows  (10)  should  be  near  p  where  p  is  the 
number  of  parameters  estimated. 

i.  The  names  of  the  independent  variables  included  in  the  model  (1 1) 


j.  B  VALUES,  the  corresponding  estimated  regression  coefficients  (12) 


l 

I 


k. 


STD  ERROR  of  the  estimates  (13) 


l.  TYPE  II  SS  (sum  of  squares)  for  each  variable  (14)  ,  which  is  the  SS  that 
is  added  to  the  error  SS  if  that  one  variable  is  removed  from  the  model. 

m.  F  values  and  PROB  >  F  associated  with  the  Type  II  sums  of  squares  (15). 


•»tn  m  parentheses  refer  to  the  sample  output. 
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B.8.2  Output  of  STKIOMSK. 

I  ;  j r  1 1  i "*■  f  i ,  H . .  * 

!  ! !  i  r .  1 1 1 , ' ' ■  0 .  ■* 

printouts  for  STKIOMSK. 
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